Introduction
FreeBSD continues to defy the rumors of its demise.
Much of the development work done this quarter was not particularly visible, especially the effort needed to ensure the upcoming 11.1 release has as few regressions as possible. Planning is also well under way for the 10.4 maintenance release which will quickly follow it.
Further work focused on moving the arm architectures' support closer to tier-1 status and improving documentation. In addition, large changes were made to the src and ports trees.
These projects and others are further detailed below.
—Mark Linimon
The deadline for submissions covering the period from July to September 2017 is October 21, 2017.
FreeBSD Team Reports
- FreeBSD Release Engineering Team
- Ports Collection
- The FreeBSD Core Team
- The FreeBSD Foundation
- The Postmaster Team
Projects
- 64-bit Inode Numbers
- Capability-Based Network Communication for Capsicum/CloudABI
- Ceph on FreeBSD
- DTS Updates
Kernel
Architectures
Userland Programs
Ports
- A New USES Macro for Porting Cargo-Based Rust Applications
- GCC (GNU Compiler Collection)
- GNOME on FreeBSD
- KDE on FreeBSD
- New Port: FRRouting
- PHP Ports: Help Improving QA
- Rust
- sndio Support in the FreeBSD Ports Collection
- TensorFlow
- Updating Port Metadata for non-x86 Architectures
- Xfce on FreeBSD
Documentation
Miscellaneous
Third-Party Projects
FreeBSD Team Reports
FreeBSD Release Engineering Team
| Links | |
| FreeBSD�11.1-RELEASE Schedule | URL: https://www.FreeBSD.org/releases/11.1R/schedule.html |
| FreeBSD Development Snapshots | URL: https://download.FreeBSD.org/ftp/snapshots/ISO-IMAGES/ |
Contact: FreeBSD�Release Engineering Team <[email protected]>
The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes, and maintaining the respective branches, among other things.
The FreeBSD�11.1-RELEASE cycle started on May 19, and continued as scheduled. FreeBSD consumers are urged to test whenever possible to help ensure the reliability and stability of the upcoming second release from the stable/11 branch.
This project was sponsored by The FreeBSD Foundation.
Ports Collection
Contact:
Ren�
Ladan
<[email protected]>
Contact: FreeBSD Ports Management Team <[email protected]>
This quarter, 2017Q2, broke the 30,000 ports landmark for the first time. The PR count is currently just under 2,500, with almost 600 of them unassigned. This quarter saw almost 7,400 commits from 171 committers. More PRs got closed this quarter than last quarter, but also more PRs got sent in, both of which are good to see.
Over the past three months, we welcomed four new committers: Bradley T. Hughes (bhughes@), Danilo G. Baio (dbaio@), Jochen Neumeister (joneum@), and Richard Gallamore (ultima@). kan@ re-joined us as a ports committer. One commit bit, that of bf@, was taken in for safekeeping after a long period of inactivity.
On the management side, the Ports Management Team welcomed back bapt@, who is working on several new features for the Ports Tree. The Ports Management Team also had its annual real-life meeting during BSDCan.
On the infrastructure side, three new USES values were introduced:
- cargo, to ease the porting of Rust packages or binaries using the cargo command (also covered separately in this report)
- groff, to handle a dependency on the groff document formatting system, that has been removed from the base system for FreeBSD 12
- meson, to provide support for projects based on Meson
The default version of PostgreSQL switched from 9.3 to 9.5, and that of Python3 from 3.5 to 3.6. The default generator for ports using cmake has been switched to ninja.
Some major version updates are: pkg 1.10.1, Firefox 54.0.1, and Chromium 59.0.3071.115.
Behind the scenes, antoine@ ran 36 exp-runs to test version updates, make the CRAN ports platform-independent, test installing bsdgrep(1) as /usr/bin/grep, test LLVM updates, test the ino64 project, and perform Makefile cleanups.
The FreeBSD Core Team
Contact: FreeBSD Core Team <[email protected]>
Core's activities during the second quarter culminated in the introduction of two new initiatives during BSDCan:
- Extending FreeBSD Project Membership
- The FreeBSD Community Process
FreeBSD Project Members
FreeBSD Project Membership being extended to more than just committers is a step that enables the Project to recognise and reward people who support us in ways other than by writing code. People that organise conferences or user groups; who are prominent supporters on social media; who triage bug reports and who test changes; and many others who contribute in various ways, are deserving of recognition for the support that they give to the Project. Core hopes that this will both encourage more people to volunteer their time and effort on behalf of the Project, and encourage those who already do to stick with the Project, if not become more deeply involved.
The naming for the new group of non-committer Project members took a few tries to get right: having tried, and rejected, "Contributor" and then "Associate", Core took the view that since what they were offerring was formal Project Membership, then that was the right thing to call it. Committers thus become those Project Members with access to commit to the Project's code repositories. Project Members receive an @FreeBSD.org e-mail address, access to various Project hardware, access to internal mailing lists and other communications channels, and invitations to attend Developer Summits in their own right. Committers in addition have commit rights in the Subversion repositories and GitHub, and active Committers can vote in Core team elections.
The FreeBSD Community Process
This is an idea that has a long pedigree within other projects, and FreeBSD is very consciously modelling its implementation on what has worked elsewhere. When a significantly disruptive or wide-scale change is proposed, we should have a formal mechanism for documenting the change and what it implies. Interested parties can then respond and the change can be evolved into the best fit for all users, or else it can be found to be impracticable and withdrawn. The documentation of the change will remain as a point of reference should the same or a similar proposal come up in the future. Creating a more formal process should help avoid endless sterile arguments about what needs to be done, without anyone feeling they have sufficient investment in the idea nor backing from the majority of the project to justify putting in the work to achieve the desired result.
The very first FCP — FCP 0 — describes the process itself. At the time of this writing, Core is voting on accepting the initial document, which can be viewed in the Project's Github repository. Two new mailing lists have been created: [email protected] is the channel for receiving notifications of new FCP proposals and discussing their content, whilst [email protected] exists to provide help with the process of drafting the FCP documents.
Other Core activities
Core is delighted to announce that Gordon Tetlow has joined the Security Officer team, and will be working on managing the Security Team caseload, freeing up other members to concentrate on the more technical aspects of vulnerability remediation. In addition, Ed Maste has joined the Security Team and is available to assist the Security Officers where necessary.
Although Florian Smeets had to step down, the postmaster team has recruited three new members and is now back up to strength.
Considering the desirability of a number of fixes that have been merged into 10-STABLE since the 10.3 release, core has approved a 10.4 release to occur shortly after the 11.1 release. This will be a normal support-lifetime release, unlike the extended lifetime of the 10.3 release, so the overall support lifetime for the 10.x branch will not be significantly extended.
During this quarter, Core has approved issuing three new commit bits. Please welcome:
- Vladimir Kondratyev (wulf@)
- Ryan Libby (rlibby@)
- Kyle Evans (kevans@)
Also, during this quarter, we had one person give up their commit bit:
- Jordan Hubbard (jkh@)
It is always unsettling when one of the Project's founding members decides to move on, but Jordan's interests have migrated away from FreeBSD-related projects and he has decided to hang up his bit once and for all.
Core would like to thank NTTA (formerly Verio) for providing hosting for a cvsup mirror for many years, and also for their kind offer to provide ongoing hosting for a machine in their Seattle facility. Since we have no need for additional North America hosting, we have declined their offer.
As usual, a number of questions have been raised about code licensing and other matters related to intellectual property. Ed Maste has registered "freebsd" on behalf of the FreeBSD Foundation on the Mastodon social media network. The "Unlicense" is suitable for code being imported into libc. We still have some code published under the old 4-clause style BSD license, where the extra clause refers specifically to the University of California. While UC has generally approved removing that clause, we need to check with all copyright holders before changing any remaining 4-clause licensing.
Core, along with the Security Team, are monitoring developments concerning the "Stack Clash" vulnerability that hit the headlines during June. Changes to the stack-guard mitigation system are underway as a response to the proof-of-concept published by Qualys.
The FreeBSD Foundation
Contact: Deb Goodkin <[email protected]>
Last quarter the Foundation was busy supporting the FreeBSD Project in so many ways! We brought on two interns from the University of Waterloo who were extremely productive, from working on a continuous integration project to adding MSDOS FAT filesystem support to makefs. We continued helping to accelerate OS changes with our internal staff of software developers, as well as funding outside software development projects, and continued promoting FreeBSD by participating in technology conferences around the world. To encourage more commercial users to donate to the Foundation, we launched a new partnership program. The FreeBSD 11.1 release effort has been led by a full-time Foundation employee, to continue keeping releases timely and reliable. Finally, we led the effort to celebrate the newly declared FreeBSD Day, to help raise awareness of FreeBSD around the world!
Below, you can read some of the highlights from our Q2 newsletter, and find writeups throughout this status report from Foundation staff members including Ed Maste, Kostik Belousov, and Glen Barber. Don't forget, we are 100% funded by donations. Please take a moment to donate now, so we can continue supporting the FreeBSD Project and community worldwide!
Q2 Development Projects Summary
Our hard work continues into the 2nd quarter of 2017. Please take a look at the highlights from our more recent Development Projects summaries.
April: FreeBSD USB Mass Storage Target Project Update
The Foundation awarded a project grant to Edward Tomasz Napierała to develop a USB mass storage target driver, using the FreeBSD CAM Target Layer (CTL) as a backend. This project allows FreeBSD on an embedded platform, such as a BeagleBone Black or Raspberry Pi Zero, to emulate a USB mass storage target, commonly known as a USB flash stick. Read more at https://www.FreeBSDfoundation.org/blog/april-2017-development-projects-update/.
May: Foundation Brings on Co-Op Students
At the beginning of May we embarked on a new path in the FreeBSD Foundation, with the hiring of co-operative education (co-op) students from the University of Waterloo. The University of Waterloo is a pioneer and leader in co-operative education, with 100% of Engineering students and a majority of Computer Science students participating in co-op programs. Read more at https://www.FreeBSDfoundation.org/blog/may-2017-development-projects-update/.
June: FreeBSD Foundation 2017 Project Proposal Solicitation (contributed by Ed Maste)
One of the ways the Foundation supports FreeBSD is by providing development grants for work on individual projects. These allow developers to propose projects they would like to undertake to improve FreeBSD and request funding to perform that work. The Foundation is always willing to receive proposals, but will occasionally issue a call for proposals to highlight specific areas of focus and to be able to collect and evaluate a group of proposals.
The proposal submission deadline was July 14, 2017, but as mentioned above, people are welcome to submit proposals at any time.
Although proposals may address any FreeBSD subsystem or infrastructure, we are particularly interested in receiving proposals related to:
- Improvements to the security of FreeBSD itself, or of applications running on FreeBSD
- New test cases, improved test infrastructure, and quality assurance
- Improved software development tools
- Projects to improve community collaboration and communication
- Improving the FreeBSD "out of the box" experience for new users on various hardware platforms
- Establishing FreeBSD as a leader in advancing projects of shared interest (such as ZFS, LLVM, or libarchive)
More details can be found at https://www.FreeBSDfoundation.org/blog/FreeBSD-foundation-2017-project-proposal-solicitation/. The full project proposal submission guidelines can be found at http://cts.vresp.com/c/?FreeBSDFoundation/d364934d4d/TEST/1b229d9af7.
Please do not hesitate to contact [email protected] with any questions.
Announcing the New Partnership Program (contributed by Deb Goodkin)
I'm excited to announce our new FreeBSD Foundation Partnership Program! Our work is 100% supported by donations from individuals and organizations. With a spending budget of $1,500,000, we rely on large donations from our commercial users to help us sustain and increase our support. Recognizing the value of these donations, and putting together a sustainable funding model, we wanted to institute benefits that highlighted this support, and recognize these donors in productive ways. Partnerships are an avenue to assist commercial users by helping them get on board more quickly with FreeBSD, share their needs with the community, and facilitate collaboration with FreeBSD developers. We believe that building these relationships with commercial users will contribute to keeping FreeBSD relevant and help provide a sustainable and healthy ecosystem.
You can check out our updated donor pages to see how we are acknowledging our Partners at https://www.FreeBSDfoundation.org/donors/. You can also find out more about this new program at https://www.FreeBSDfoundation.org/FreeBSD-foundation-partnership-program/.
When I was in China last week, I had a chance to talk to a few companies about our new partnership program, and it definitely generated more interest in supporting our efforts.
We are continuing to reach out to commercial users for help that will enable us to provide more outreach and support for FreeBSD. This includes funding more projects to improve FreeBSD, providing FreeBSD education and training, and recruiting more contributors to the Project. We can only provide the above support with your donations, and we need your help to connect us with your companies. Please consider notifying your organization about our new Partnership Program and helping to connect us with the appropriate contacts at your company.
Your donations will help us:
- Accelerate improvements and add new features to FreeBSD
- Support release engineering efforts full-time
- Create and provide FreeBSD educational and training material
- Provide face-to-face opportunities for developers to work together
- Improve and support FreeBSD infrastructure
We need your support to continue improving FreeBSD.
Q2 2017 Conference Recaps
From sponsoring events to attending conferences, the Foundation continued its mission of advocacy in the second quarter of 2017. Over the past few weeks, members of the Foundation team represented the Project and the Foundation at events around the world. Below are just a few of the conference recaps.
FOSSASIA 2017 (contributed by Philip Paeps)
The Foundation kindly funded part of my travel from Tokyo to Singapore to attend FOSSASIA. I gave the "FreeBSD is not a Linux Distribution" presentation that Foundation board member George Neville-Neil wrote for Open Source China in December. My presentation was well-attended, and I got a lot of good questions from the primarily Linux-oriented audience. Read more at https://www.FreeBSDfoundation.org/blog/fossasia-2017-trip-report-philip-paeps/.
OSCON 2017 (contributed by Ed Maste)
I represented the FreeBSD Foundation at OSCON 2017, which took place May 8-11, 2017, in Austin, TX: https://conferences.oreilly.com/oscon/oscon-tx.
The Foundation booth was also staffed by FreeBSD committer Brad Davis and Doug Mcintire from Netgate. We met up Wednesday morning to set up the table. We were part of a "nonprofit pavilion" which consisted of eight or so tables, located between Open Camps and Operation Code.
To help attract booth traffic, I brought a Raspberry Pi 3, with a small LCD display attached. As a demo, the Raspberry Pi showed a video of a Gource rendering of changes to the FreeBSD source tree over time (see example at https://www.youtube.com/watch?v=vZ8Sspua0Ks). Read more at https://www.FreeBSDfoundation.org/blog/conference-recap-oscon-2017/.
Rootconf 2017 (contributed by Philip Paeps)
In mid-May I presented at Rootconf 2017 in Bangalore. Rootconf is India's principal conference where systems and operations engineers share real-world knowledge about building reliable systems: https://rootconf.in/2017/.
As always, it was interesting to hear the difficulties people face trying to run reliable systems on less reliable platforms. While many of the presentations were very Linux-specific and not very exciting to me, a couple of talks did catch my eye.
I particularly enjoyed the talk by Aruna Sankaranarayanan (https://www.youtube.com/watch?v=XQJ7YhVoSWI&feature=youtu.be) explaining how Mapbox takes advantage of Amazon's "spot pricing" mechanism by spawning and shutting down machines at different price points to optimize for cost without compromising availability. Their spotswap https://github.com/mapbox/spotswap/ software has been released under a BSD license. It sounds as though it should be possible to port this to FreeBSD with minimal effort. Read more at https://www.FreeBSDfoundation.org/blog/rootconf-2017-trip-report-philip-paeps/.
BSDCan 2017/FreeBSD Developers Summit (contributed by Deb Goodkin)
One of our initiatives is to assist in providing face-to-face knowledge sharing and development opportunities around the world. One way we do this is by sponsoring BSD-related conferences and FreeBSD Developer and Vendor Summits. We recently sponsored both BSDCan 2017 and the FreeBSD Developer and Vendor Summit in Ottawa, Ontario, Canada, which took place June 7-10, 2017. Many of our board and staff members attended the summit and conference to run tutorials, give presentations, lead sessions, work with developers, give demos, and share knowledge.
In addition, this year we were pleased to bring our new University of Waterloo interns to the conference where they had the opportunity to demonstrate some of their projects at the Foundation table. Read more at https://www.FreeBSDfoundation.org/blog/conference-recap-bsdcan-2017FreeBSD-developers-summit/.
Open Travel Grant Applications
The Foundation recognizes the importance of bringing members of the FreeBSD community face-to-face to both further development of the Project and spread the word about FreeBSD. Travel grants are available to community members who need assistance with travel expenses for attending conferences related to FreeBSD development and advocacy. Please note: the travel grant policy has been recently updated. Please carefully review it before submitting your application.
More information about travel grants is available at: https://www.FreeBSDfoundation.org/what-we-do/grants/travel-grants/.
FreeBSD Day was June 19! (contributed by Anne Dickison)
June 19th was declared FreeBSD Day! Thank you to everyone who joined us in honoring the FreeBSD Project's pioneering legacy and continuing impact on technology. Find out more about FreeBSD Day and how we celebrated here at https://www.FreeBSDfoundation.org/blog/happy-FreeBSD-day/.
Upcoming Events
Find out about upcoming Foundation events at https://www.FreeBSDfoundation.org/news-and-events/upcoming-events/.
FreeBSD Journal
The May/June 2017 Issue of the FreeBSD Journal is now available. Don't miss articles on FreeBSD's Firewall Feast, CADETS: Blending Tracing and Security on FreeBSD, Toward Oblivious Sandboxing with Capsicum, and more. (https://www.FreeBSDfoundation.org/past-issues/security/)
Did you miss the March/April issue? Check out articles on CFEngine, Puppet on FreeBSD, Vagrant, and more! (https://www.FreeBSDfoundation.org/past-issues/configuration-management/) As a recent addition of functionality, browser-based subscribers now have the ability to download and share PDFs of the articles!
Sample Issue! If you've ever wanted to read through an entire issue of the FreeBSD Journal, now's your chance. Download the sample issue from https://mydigitalpublication.com/publication/?i=296880#{"issue_id":296880,"numpages":1,"page":1} and be sure to share with your friends and colleagues. Not a subscriber? Sign up today at https://www.FreeBSDfoundation.org/journal/.
More information about the Foundation's doings and goings-on can be found in our own quarterly newsletter, linked above.
The Postmaster Team
| Links | |
| The Postmaster Team | URL: https://www.FreeBSD.org/administration.html#t-postmaster |
Contact:
David
Wolfskill
<[email protected]>
Contact:
Larry
Rosenman
<[email protected]>
Contact:
Ryan
Steinmetz
<[email protected]>
Contact:
Eygene
Ryabinkin
<[email protected]>
Contact:
Remko
Lodder
<[email protected]>
Contact:
Kurt
Jaeger
<[email protected]>
Postmaster handles the mail flow for the FreeBSD project.
Clusteradm provides us with four jails: mailman, mailarchive, mx1, and mx2. In addition, there is some part of the setup running on freefall.FreeBSD.org. The system uses postfix, mailman, spamassassin, and some other tools from the ports tree to handle the mail flow. We use a very small, non-public Subversion repository for parts of the configuration.
During Q2, Larry Rosenman, Kurt Jaeger, Eygene Ryabinkin, Remko Lodder and Ryan Steinmetz joined the Postmaster Team, and Florian Smeets left the Postmaster Team.
Thanks to Florian for his long service in that role! David Wolfskill is planning to leave the role as soon as the new team members are settled. Vsevolod Stakhov plans to provide us with support to integrate rspamd into the setup, as well.
The workload for the Postmaster Team is not high, but the complexity of the setup has its own demands.
Open tasks:
- We need to improve our internal documentation of workflows and processes.
- We should consider adding some monitoring to provide quarterly numbers on the mail flow.
Projects
64-bit Inode Numbers
| Links | |
| Phabricator Review | URL: https://reviews.FreeBSD.org/D10439 |
Contact:
Gleb
Kurtsou
<[email protected]>
Contact:
Konstantin
Belousov
<[email protected]>
Contact:
Kirk
McKusick
<[email protected]>
The 64-bit inode project was completed and merged into FreeBSD�12 on May 23, 2017. It extends the ino_t, dev_t, and nlink_t types to be 64-bit integers. It modifies the struct dirent layout to add a d_off field, increases the size of d_fileno to 64 bits, increases the size of d_namlen to 16 bits, and changes the required alignment of the structure. It increases the struct statfs f_mntfromname[] and f_mntonname[] array lengths from MNAMELEN to 1024.
ABI breakage is mitigated by providing compatibility using versioned symbols, ingenious use of the existing padding in structures, and employing various other tricks. Unfortunately, not everything can be fixed, especially outside the base system. For instance, third-party APIs which pass struct stat as parameters are broken in backward- and forward-incompatible ways.
The ABI for kinfo-consuming sysctl MIBs is changed in a backward-compatible way, but there is no general mechanism to handle other sysctl MIBS which return structures where the layout has changed. In our consideration, this breakage is either in management interfaces, where we usually allow ABI slippage, or is not important.
The layout of struct xvnode changed, and no compatibility shims are provided.
For struct xtty, the dev_t tty device member was reduced to be just uint32_t. It was decided that maintaining ABI compatability in this case is more useful than reporting a 64-bit dev_t value, for the sake of pstat.
Updating note: strictly follow the instructions in UPDATING. Build and install the new kernel with the COMPAT_FREEBSD11 option enabled, then reboot, and only then install the new world.
Credits: The 64-bit inode project, also known as ino64, started life many years ago as a project by Gleb Kurtsou (gleb). Kirk McKusick (mckusick) then picked up and updated the patch, and acted as a flag-waver. Feedback, suggestions, and discussions were carried out by Ed Maste (emaste), John Baldwin (jhb), Jilles Tjoelker (jilles), and Rick Macklem (rmacklem). Kris Moore (kris) performed an initial ports investigation followed by an exp-run by Antoine Brodin (antoine). Essential and all-embracing testing was done by Peter Holm (pho). The heavy lifting of coordinating all these efforts and bringing the project to completion were done by Konstantin Belousov (kib).
This project was sponsored by The FreeBSD Foundation (emaste, kib).
Capability-Based Network Communication for Capsicum/CloudABI
| Links | |
| ARPC: GRPC-Like RPC Library That Supports File Descriptor Passing | URL: https://github.com/NuxiNL/arpc |
| Flower: A Label-Based Network Backplane | URL: https://github.com/NuxiNL/flower |
Contact: Ed Schouten <[email protected]>
One of the weaknesses of Capsicum and CloudABI is that it is not easy to develop applications that need to make outgoing network connections, since system calls like connect() and sendto() are disabled. Though we can sometimes work around this by ensuring that the sandboxed process already possesses socket file descriptors on startup, this does not allow the destination process to be restarted, moved to a different network address, be load balanced, etc..
Coming up with a solution for this is quite important for me, as I am currently working on making CloudABI work on top of Kubernetes, Google's open source cluster management suite. The idea is that Kubernetes will schedule CloudABI processes instead of Docker containers. All of these CloudABI processes will have their dependencies on other services in the cluster injected explicitly, making internal communication very secure. All of this is intended to work on FreeBSD as well, of course!
To solve this problem, I've been working on a daemon called Flower (read: flow-er) that allows software to register services and connect to them. Servers are identified by a set of labels with values (e.g., {datacenter: 'frankfurt', service: 'mysql'}). Clients can connect these servers by providing the corresponding label(s). Flower's security model is capability-based, just like Capsicum. The ability to bind and connect can be limited by permanently constraining labels to certain values.
Flower has been designed not to act as a proxy. It does not copy any data. It merely forwards existing socket file descriptors or creates UNIX socket pairs and hands these out to its clients and servers. To realize this, processes communicate with Flower using an RPC library called ARPC. ARPC is a very simple clone of Google's GRPC, with the special feature that messages (Protobufs) can have file descriptors attached.
This project was sponsored by Nuxi, the Netherlands.
Open tasks:
- Finish implementing the Flower code.
- Integrate Flower with the Kubernetes/CloudABI runtime.
- Release the Kubernetes/CloudABI runtime as open source software.
Ceph on FreeBSD
| Links | |
| Ceph Main Site | URL: http://ceph.com |
| Main Repository | URL: https://github.com/ceph/ceph |
| My FreeBSD Fork | URL: https://github.com/wjwithagen/ceph |
Contact: Willem Jan Withagen <[email protected]>
Ceph is a distributed object store and file system designed to provide excellent performance, reliability and scalability.
Object Storage
Ceph provides seamless access to objects using native language bindings or radosgw, a REST interface that is compatible with applications written for S3 and Swift.
Block Storage
Ceph's RADOS Block Device (RBD) provides access to block device images that are striped and replicated across the entire storage cluster.
File System
Ceph provides a POSIX-compliant network file system that aims for high performance, large data storage, and maximum compatibility with legacy applications.
I started looking into Ceph because the HAST solution with CARP and ggate did not really do what I was looking for. I aim to run a Ceph storage cluster of storage nodes that are running ZFS, with user workstations running bhyve on RBD disks that are stored in Ceph.
Compiling for FreeBSD will now build most of the tools available in Ceph.
The most important changes since the last report are:
- Ceph has released release candidate v12.1.0 (aka Luminous); the corresponding packaging is sitting in my tree waiting for Luminous to be actually released.
- ceph-fuse works, and allows mounting of cephfs filesystems. The speed is not impressive, but it does work.
- rbd-ggate is available to create a Ceph rbd backed device. rbd-ggate was submitted by Mykola Golub. It works in a rather simple fashion: once a cluster is functioning, rbd import and rbd-ggate map are used to create ggate-like devices backed by the Ceph cluster.
Other improvements since the previous report:
- Some bugs in the init-ceph code (needed for rc.d) are being fixed.
- RBD and rados are functioning.
- The needed compatability code was written so that FreeBSD and Linux daemons can operate together in a single cluster.
- More of the awkward dependancies on Linux-isms are deleted — only /bin/bash is there to stay.
The next forthcoming official release of Ceph is called Luminous (v12.1.0). As soon as it is available from upstream, a port will be provided for FreeBSD.
To get things running on a FreeBSD system, run pkg install net/ceph-devel or clone https://github.com/wjwithagen/ceph, check out the wip.freebsd.201707 branch, and build manually by running ./do_freebsd.sh in the checkout root.
Parts not (yet) included:
- KRBD — but rbd-ggate is usable in its stead.
- BlueStore — FreeBSD and Linux have different AIO APIs, and that incompatibility needs to be resolved somehow. Additionally, there is discussion in FreeBSD about aio_cancel not working for all device types.
Open tasks:
- Run integration tests to see if the FreeBSD daemons will work with a Linux Ceph platform.
- Investigate the keystore, which can be embedded in the kernel on Linux and currently prevents building Cephfs and some other parts. The first question is whether it is really required, or if only KRBD requires it.
- Scheduler information is not used at the moment, because the schedulers work rather differently between Linux and FreeBSD. But at a certain point in time, this will need some attention (in src/common/Thread.cc).
- Improve the FreeBSD init scripts in the Ceph stack, both for testing purposes and for running Ceph on production machines. Work on ceph-disk and ceph-deploy to make it more FreeBSD- and ZFS-compatible.
- Build a test cluster and start running some of the teuthology integration tests on it. Teuthology wants to build its own libvirt, and that does not quite work with all the packages FreeBSD already has in place. There are many details to work out here.
- Design a virtual disk implementation that can be used with bhyve and attached to an RBD image.
DTS Updates
Contact: Emmanuel Vadot <[email protected]>
DTS (Device Tree Source) files provide a human-readable source description of the hardware resources for a given computer system (such as ARM- or MIPS-based embedded boards). The DTS source representation must be compiled into a binary format in order to be linked into the kernel and used to locate devices at runtime.
The DTS files in FreeBSD were updated to match the versions from Linux 4.11, to represent more modern devices and provide more accurate representations.
Kernel
Coda revival
| Links | |
| GitHub Repository | URL: https://github.com/trasz/FreeBSD/tree/coda |
Contact: Edward Tomasz Napierała <[email protected]>
Coda is a distributed file system developed as a research project at Carnegie Mellon University, descended from a older version of the Andrew File System. It got dropped from FreeBSD some five years ago, due to not having been adopted for a MPSAFE world. The focus for this current project is to bring it back into sufficiently workable shape that it could return to the kernel. It is currently in a working condition. Work is underway to test it better, fix whatever issues are found, and commit it to 12-CURRENT.
This project was sponsored by Chalmers University of Technology.
Open tasks:
- Additional testing.
- Update the userspace components (net/coda_client and net/coda_server).
FreeBSD Driver for the Annapurna Labs ENA
| Links | |
| Enhanced Networking Guide | URL: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html |
Contact:
Marcin
Wojtas
<[email protected]>
Contact:
Michał
Krawczyk
<[email protected]>
The ENA (Elastic Network Adapter) is a 25G SmartNIC developed by Annapurna Labs and is based on a custom ARMv8 chip. This is a high-performance networking card available in the AWS offerings. It introduces enhancements in network utilization scalability on EC2 machines under the control of various operating systems, in particular FreeBSD.
The goal of FreeBSD enablement is to provide top performance and a wide range of monitoring and management features such as:
- multiple queue modes
- hardware offloads (rx and tx checksum)
- an admin queue
- asynchronous notifications
- robust hardware access
- a scalable number of MSI-X vectors
- hardware counters
- watchdog mechanism
- LRO
- RSS
The driver is available in the kernel source tree as of r318647.
This project was sponsored by Annapurna Labs — an Amazon company.
Open tasks:
- Add RSS configuration from userspace (via sysctls).
- Add support for LLQ mechanisms.
Intel 10G Driver Update
| Links | |
| Commit Adding X553 ix/ixv Support | URL: https://reviews.FreeBSD.org/D11232 |
Contact:
Chris
Galazka
<[email protected]>
Contact:
Jeb
Cramer
<[email protected]>
The ix and ixv network interface drivers support a variety of Intel network interfaces, with line speeds at 10 Gbit/second.
This quarter, the drivers gained support for the X553 network interface, which is found on System-on-a-Chip devices based on the Denverton platform. This update should allow FreeBSD to be more useful on a new class of hardware platform.
Work is also underway to convert these drivers to use the iflib network driver library, which should ease future maintenance of the drivers, as well as the network subsystem as a whole.
pNFS Server Plan B
| Links | |
| Testing Instructions | URL: http://people.FreeBSD.org/~rmacklem/pnfs-planb-setup.txt |
Contact: Rick Macklem <[email protected]>
Parallel NFS (pNFS) is an extension to the NFSv4 protocol that allows for file accesses within a single logical mount to be performed against multiple file servers, with the potential for data access to occur in parallel. The pNFS "layout" in use specifies how the division occurs, with metadata operations occurring against the main server, and bulk data operations (read/write/setattr/etc.) occurring via a layout-specific scheme between the client and the data servers.
My first attempt at a pNFS server using GlusterFS was a dud. It worked, but performance was so poor that it was not usable. This attempt that I call "Plan B", only uses FreeBSD, with one FreeBSD server handling the metadata operations and multiple FreeBSD servers configured to serve data, is now ready for third-party testing. If testing by third parties goes well, I anticipate the code will be merged into FreeBSD head in time for FreeBSD�12. Fairly recent FreeBSD or Linux systems should be usable as pNFS clients for testing. This server supports the File Layout, which is supported by both of these clients.
There is no support for the Flex Files Layout or mirroring at this time. I hope to use the Flex Files Layout to add mirroring support over the next year or so. Striping is not supported, and I have no plans for implementing this at the moment.
The patched FreeBSD sources may now be accessed for testing via either Subversion or download of a gzipped tarball. They consist of a patched kernel and nfsd and can be used on any FreeBSD�11 or later system.
Open tasks:
- Testing by others will be needed, now that the code is available.
Architectures
FreeBSD on Marvell Armada38x
Contact:
Marcin
Wojtas
<[email protected]>
Contact:
Zbigniew
Bodek
<[email protected]>
Work proceeds to finalize the process of bringing support for the Marvell Armada38x platform into FreeBSD head.
The most important parts of the recent effort are:
- Add the network driver (NETA)
- Enable coherent busdma operation for all ARMv7 SoCs
- Add various low-level optimizations, such as L1 cache prefetch and MBUS quirks
- Enable PL310 L2 cache controller
- Add SDHCI support
- Fixes for the e6000sw driver and a rework of its PHY handling
- Support multi-port PCIe operation
- Various fixes and enhancements of the common Marvell code
- Fix and enable support for performance counters (HWPMC)
This project was sponsored by Stormshield, Semihalf, and Netgate.
FreeBSD/arm64
| Links | |
| FreeBSD arm64 Wiki Page | URL: https://wiki.FreeBSD.org/arm64 |
Contact: Andrew Turner <[email protected]>
Support for the Privilege Access Never (PAN) feature was added. This stops the kernel from accessing userspace memory, except through specific instructions. This helps security by only allowing access to userspace via the correct accessor functions. This is enabled on all supported CPUs that implement ARMv8.1 or later.
The pmap code now supports the Unprivileged execute-never (UXN) and Privileged execute-never (PXN) bits in the page tables. These bits stop userspace and the kernel, respectively, from executing instructions on any marked page.
The performance of the pmap layer has been improved. Many of the cache handling function calls have been removed. Some were needed early on to work around other bugs that have now been fixed. The removal of these calls has led to a large performance improvement.
The kernel now uses crc32c instructions where appropriate. These are an optional set of instructions to perform crc32c checksumming quickly without using a lookup table.c
The VM_MEMATTR_WRITE_THROUGH memory attribute is now supported. This is used to allocate memory for the framebuffer. Previously, the kernel would use cached memory; however, this leads to visual artifacts. The write-through flag fixes these by writing data out to RAM.
The default linker on arm64 is now lld. This means that FreeBSD is able to build itself with just the components in the base system, a big milestone!
Userland Programs
DTC
Contact: Emmanuel Vadot <[email protected]>
The in-tree DTC (Device Tree Compiler) was switched to use the BSD-licensed version by default. (The previous default DTC is licensed under the GPL.) The current version supports overlays and is able to compile every DTS (Device Tree Source) used by the FreeBSD arm releases. The ports GPL version was updated to the latest release (1.4.4). The in-tree GPL version is still present but the goal is to remove it before FreeBSD 12.0.
Using LLVM's LLD Linker as FreeBSD's System Linker
Contact:
Rafael
Esp�ndola
<[email protected]>
Contact:
Ed
Maste
<[email protected]>
LLD is the linker in the LLVM family of projects. It is a high-performance linker that supports the ELF, COFF and Mach-O object formats. It is broadly compatible with the common linkers used for each file format. For ELF this is the GNU Binary File Descriptor (BFD) ld and GNU gold. However, LLD's authors are not constrained by strict compatibility where it would hamper performance or desired functionality.
LLD is now used as the default system linker for FreeBSD/arm64 and can link a working kernel, kernel modules, and userland for FreeBSD/amd64. LLD can also link a working kernel and modules (but not userland) for FreeBSD/arm and FreeBSD/i386.
Work is ongoing to address ports that do not build with LLD as the system linker (either by fixing the port, or configuring the port to be linked by GNU ld).
For FreeBSD�12.0 we expect to use LLD as the system linker for the same set of architectures that use Clang by default: 32- and 64-bit arm and x86.
This project was sponsored by The FreeBSD Foundation.
Open tasks:
- Fix libtool to detect LLD and pass the same command line arguments as for GNU ld and gold.
- Investigate the remaining amd64 and arm64 port build failures.
- Investigate and improve LLD on i386 and arm, before the creation of the stable/12 branch.
- Investigate and improve LLD on all other architectures.
- Extensive testing.
Ports
A New USES Macro for Porting Cargo-Based Rust Applications
Contact: Tobias Kortkamp <[email protected]>
Support in the Ports Collection for applications written in the Rust programming language that use Rust's package manager Cargo was added, via a new USES=cargo setting. The work is based on the cargo module from the OpenBSD ports tree.
This should significantly ease the porting of Rust applications, as previously porters had to create their own tarball of the application's dependencies or find other manual ways of bringing them in.
Several new ports were added that use it, for example:
- Alacritty, a GPU-accelerated terminal emulator
- Exa, a modern replacement for ls
- Ripgrep, a line-oriented search tool that combines the usability of The Silver Searcher with the raw speed of GNU grep
Open tasks:
- Add documentation for the new feature.
GCC (GNU Compiler Collection)
Contact:
Gerald
Pfeifer
<[email protected]>
Contact:
Andreas
Tobler
<[email protected]>
The default version of GCC in the Ports Collection (the one requested by USE_GCC=yes and various USES=compiler invocations) has been updated from GCC 4.9.4 to GCC 5.4.
This new major version brings many new capabilities and improvements, as well as some changes that may require adjustments. The latter category includes many new compiler warnings, significant improvements to inter-procedural optimizations, and link-time optimization.
The default mode for C is now -std=gnu11 instead of -std=gnu89. The C++ front end has full C++14 language support, including C++14 variable templates, C++14 aggregates with non-static data member initializers, C++14 extended constexpr, and more. The Standard C++ Library (libstdc++) has full C++11 support and experimental full C++14 support. It uses a new ABI by default.
The lang/gcc port now is a meta-port that pulls in the respective lang/gccX port (based on the setting of $GCC_DEFAULT) and defines gcc, g++, and gfortran as symlinks to the respective versioned binaries.
This is the end of a long journey establishing this infrastructure, which is now similar that used by the python ports, for example. Having the new infrastructure makes upgrading the default, as well as locally adjusting the default version, a lot easier.
gcc8-devel has been added, and armv6hf support removed, and we made adjustments for newer versions of FreeBSD. Also of note are various cleanups and changes to improve the robustness of our packages and the addition of support for aarch64 to many ports.
Thanks to dim@, jbeich@, tijl@, mat@, miwi@, linimon@ for assisting with this work.
Open tasks:
- The update of the default version of GCC from GCC 5.4 to GCC 6.4 is stalled, unfortunately. The work on the GCC and insfrastructure sides is complete, but unfortunately there are a number of broken ports that need to be adjusted/fixed. Any help is very appreciated; see PR 219275 for details.
GNOME on FreeBSD
Contact: FreeBSD GNOME Team <[email protected]>
The FreeBSD GNOME Team maintains the GNOME, MATE, and CINNAMON desktop environments and graphical user interfaces for FreeBSD. GNOME 3 is part of the GNU Project. MATE is a fork of the GNOME 2 desktop. CINNAMON is a desktop environment using GNOME 3 technologies but with a GNOME 2 look and feel.
After a period of not much activity, this quarter we started a little experiment in how we merge ports from the development repo to the FreeBSD Ports Collection. Instead of merging everything in one big commit, we have been updating the GNOME ports one at a time or in small groups. For example, the GTK+ stack and the Evolution Suite were updated as groups, and all the gnome-games components were done in one commit. It might be a bit more work preparing and testing the updates, but on the plus side, it easy to keep track of what is going on, and allows us to pay attention to the details. It should also make it easier to commit smaller changes.
This quarter started with the update of GTK+ 3 to 3.22.15, and the underlying libraries to their latest stable versions. After the GTK+ update, work started on getting newer versions of other GNOME applications updated.
The webkit2-gtk3 port was first updated to the 2.14 series and later to 2.16.3, which is the latest stable version. This step was needed because 2.16 couldn't be built on FreeBSD�10.3 without some required framework changes.
harfbuzz-icu was split off from the main harfbuzz port. This drops the heavy icu dependency from the main harfbuzz port.
A longstanding GLib/gio bug was fixed that had previously caused crashes of gnome-shell and other applications when share/applications was modified, as happens on pkg install or deinstall.
Many of these updates are based on work previously done in the Gnome development branch by Ruslan Makhmatkhanov, Gustau Perez and Koop Mast.
Open tasks:
- Porting of Mutter/Gnome-shell/GDM 3.24 is complete. Unfortunately, GDM is blocking the update because of a "handoff" bug to the session after login.
- Fix the printer submenu in gnome-control-center. As a workaround, system-config-printer can be used to configure printers.
- MATE 1.18 is being QA tested and should arrive in early July.
KDE on FreeBSD
Contact: KDE on FreeBSD Team <[email protected]>
The KDE on FreeBSD team focuses on packaging KDE and Qt, and making sure that their experience on FreeBSD is as good as possible.
This quarter, in addition to the regular updates to the KDE, Qt, and related ports, there have also been some changes behind the scenes: our development repository has moved to GitHub, and FreeBSD is now part of KDE's official continuous integration (CI infrastructure).
After the X.Org and GNOME ports teams, the KDE on FreeBSD team has moved its development repository to GitHub. This should make it easier for others to collaborate with us via pull requests, and by basing all our changes on top of the official ports tree we also hope this reduces the amount of conflicts and churn we need to deal with when landing big updates across the tree. We would like to thank iXsystems for hosting and supporting our area51 Subversion repository for many years.
FreeBSD has finally joined KDE's CI (Continuous Integration) system as a tier-1 platform. KDE CI builds all the KDE sources — 70 frameworks, the KDE Plasma Desktop and a plethora of KDE Applications — continuously, straight from KDE's git repositories. There is strong commitment from upstream and the downstream KDE-FreeBSD team to reduce the amount of patching in the KDE ports to as little as possible. The first effects are being felt in expanding the set of unit tests to include FreeBSD-specific situations, and in extending Qt to handle FreeBSD filesystems better. In addition to the KDE sysadmins, we would also like to extend our thanks to Adriaan de Groot, who is both a KDE committer and part of our KDE on FreeBSD team, for spearheading these efforts.
The following big updates landed in the ports tree this quarter:
- CMake was updated to 3.8.0 and 3.8.2
- KDE Frameworks was updated to 5.33, 5.34 and 5.35
- The Calligra office suite was updated to 3.0.1, the first release in the ports tree to be based on KDE Frameworks 5, and the latest stable release upstream
- The Konversation IRC client was updated to 1.7.2, the latest upstream release and the first ports version based on KDE Frameworks 5
- KchmViewer was updated to 7.7, which is based on KDE Frameworks 5
- LabPlot was updated to 2.3.0 and 2.4.0, and is now based on KDE Frameworks 5
- QtCreator was upated to 4.2.2 and subsequently to 4.3.0
- py-sip was updated to 4.19.2, PyQt4 to 4.12 and PyQt5 to 5.7.1
- Several fixes for ARMv6 landed in the Qt4 and Qt5 ports — thanks to Mika�l Urankar
After several review rounds and exp-runs, Tobias Berner (tcberner@) finally made the Ninja generator the default for CMake-based ports, so that devel/ninja is used instead of (g)make in most cases. This should make most builds faster, even if only by a small margin. Adriaan de Groot also wrote a blog post about the change.
New Port: FRRouting
| Links | |
| FRRouting Home Page | URL: https://frrouting.org/ |
Contact: Olivier Cochard-Labb� <[email protected]>
FRRouting (FRR), a Quagga fork, is an IP routing protocol suite for Linux and Unix platforms which includes protocol daemons for BGP, IS-IS, OSPF and RIP (LPD and PIM support need to be fixed on FreeBSD). FRR is a Linux Foundation Collaborative Project with contributors including 6WIND, Architecture Technology Corporation, Big Switch Networks, Cumulus Networks, LabN Consulting, NetDEF (OpenSourceRouting), Orange, Volta Networks, and other companies.
This project was sponsored by Orange.
PHP Ports: Help Improving QA
| Links | |
| My Patreon Page | URL: https://www.patreon.com/TorstenZuehlsdorff |
Contact: Torsten Z�hlsdorff <[email protected]>
As maintainer of the PHP ports, I first want to thank you all for the great feedback and patches I receive, in many forms. You keep my life interesting!
In the past few months I learned a lot about various configurations, settings and bugs. Also, sadly, there are always PRs, patches and emails left unanswered, because of missing time on my side.
I want to improve the situation by adding more automatic QA testing, but I need help to do so. Please send me your non-standard PHP-configurations or describe your exotic setups! These can be as simple as changed default versions, like LibreSSL instead of OpenSSL or the GCC version used for compiling. I, for example, always use another PostgreSQL-version than the default (and always PHP 7.1). Of course, this also covers port options set in an non-default way or setups that change variables to allow for multiple PHP installations, etc..
I plan to test on all supported FreeBSD versions, so you only need to mention if you are using an unsupported version.
Note: Since PHP 7.2 is coming (hopefully on schedule), I will test PHP 7.2 from the onset with all the provided configurations, too.
Open tasks:
- Document the various configurations to be tested.
- Setup the automatic QA infrastructure.
Rust
Contact: FreeBSD Rust team <[email protected]>
Rust was updated to 1.18.0 and Cargo to 0.19.0, the latest versions at the time of this writing.
lang/rust was enabled on FreeBSD/aarch64 and work has continued on devel/cargo to achieve the same. We are also making slow progress to add support for even more platforms.
Discussion has started upstream to support API/ABI-breaking changes between major releases of operating systems. For instance, this is required to be able to target both FreeBSD 11.x and 12.x, which have ABI changes involving important structures. Once support is added upstream, it will be possible to target a specific ABI and do cross-compilation.
lang/rust-nightly was marked as broken for now. We need to revisit how the port is built so we can use the x.py script as recommended by upstream.
Tobias Kortkamp (tobik@) created the USES=cargo setting to make it easy to add Rust applications to the Ports Collection. This is further detailed in a separate entry in this quarterly status report.
The compiler, rustc, is crashing sometimes when there is a compilation error. Therefore, there is a bit of work to do to improve its stability.
There is some code duplication between the lang/rust* and devel/cargo Makefiles. These all deserve a bit of cleanup, and it might be useful to create a USES=rust Makefile helper.
Open tasks:
- Bootstrap Rust on more platforms.
- Investigate compiler crashes.
- Investigate how to speed up lang/rust* compilation times.
sndio Support in the FreeBSD Ports Collection
Contact: Tobias Kortkamp <[email protected]>
sndio is a small audio and MIDI framework that is part of the OpenBSD project. It provides a lightweight audio and MIDI server, sndiod. It currently supports OpenBSD, FreeBSD, DragonFly BSD, and Linux.
The porting effort to FreeBSD and OSS started last year and the sndio backend support in the FreeBSD Ports Collection can now be considered good enough for daily use.
Sndio offers network transparency through sndiod, which provides an easy way to share your audio devices with other machines/VMs/jails on your network. However, applications and libraries need to support playing and recording through it. To that end, I submitted several patches to various ports over the course of the last year.
Here's a short selection of ports that now support sndio in the FreeBSD Ports Collection:
- Most games, via audio/openal-soft, devel/sdl12, and devel/sdl20.
- GStreamer-based applications and WebKit-based browsers through two new GStreamer plugins (audio/gstreamer1-plugins-sndio and audio/gstreamer-plugins-sndio).
- Firefox, Firefox ESR, Seamonkey, Chromium, and Iridium. The browsers currently lack or have a non-functional OSS backend. Sndio support provides a BSD-native alternative to the ALSA and PulseAudio backends.
- Video players like VLC, Totem, mpv, mplayer, etc..
- Audio players like Clementine, cmus, mpd, mpg123, siren, xmp, etc..
- SoX.
- Shairport Sync, through a newly implemented backend.
- JACK.
- PulseAudio, through audio/pulseaudio-module-sndio.
Open tasks:
- Commit a backport of Kodi's new sndio backend to the Ports Collection.
- If you maintain or use an audio-related port, consider checking whether it includes an sndio backend, and adding an SNDIO option. Thanks to the OpenBSD developers, several open-source projects already include one, so adding it might be very easy to do.
TensorFlow
Contact: Jov <[email protected]>
As described on its website, "TensorFlow™ is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API. TensorFlow was originally developed by researchers and engineers working on the Google Brain Team within Google's Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well."
TensorFlow now is the most popular platform/library for machine learning and AI. There are official binaries for Linux, Mac, Windows, and Android, but no official support for FreeBSD. For the last several months, I have done some work to make TensorFlow available on FreeBSD. Some notable items include:
- bazel was patched to not depend on /proc at build time. bazel is a build tool made by Google. It uses /proc to get path-to-self when building C++ code, but mounting /proc is usually not allowed when building as an unprivileged user.
- TensorFlow can now be built on FreeBSD�10.x by using clang38 as the default bazel cross-build tool.
- Patch the bazel workspace files to allow TensorFlow to be built using offline third-party dependencies. This work is needed because the FreeBSD Ports framework does not allow network access except during the fetch stage.
- Fix the build on FreeBSD i386.
- Make TensorFlow build with either Python 2 or Python 3.
- Update to the latest version, which is tensorflow-1.2.0.
TensorFlow can now be run on FreeBSD in CPU-only mode. Some functional tests have been performed on some combinations of FreeBSD 10.3-RELEASE and 11.0-RELEASE, amd64 and i386, and Python 2.7 and Python 3.6.
This port would not be possible without substantial assistance from bapt@, lwhsu@, mat@, and koobs@ — thank you for your advice, review, and help! You are very nice and I learned a lot about FreeBSD and the Ports framework from you.
Open tasks:
- Review, test, comment, and most importantly, commit to the Ports Collection.
- Fix OpenCL (GPU acceleration) support on FreeBSD.
- Port tensorflow-serving, which is a flexible, high-performance serving system for machine learning models produced by TensorFlow.
- Set up a CI for TensorFlow on FreeBSD and give early notice to upstream when they break TensorFlow on FreeBSD.
Updating Port Metadata for non-x86 Architectures
Contact: Mark Linimon <[email protected]>
I have been analyzing the error logs from ports builds for all non-x86 architectures, including both the logs published on the package build cluster and also other builds of powerpc64 and sparc64.
From this analysis, I have marked almost all the failing ports as either BROKEN or NOT_FOR/ONLY_FOR, as appropriate.
The intent of this work is not to make life harder for anyone, but rather, in fact, the opposite. With these definitions in place, it is possible to scan the poudriere bulk build output (the "Ignored ports" portion, in particular) and see quickly what ports are failing to build and why. Previously, finding the exact reason why a build failed needed some research (portsmon only analyzes failure messages on amd64). Additionally, it is extremely difficult to work through several hundred logs that simply say "failed to compile", "failed to link", and so forth.
This is part of an effort to identify where we need further work to bring sufficient Ports Collection support to, e.g., armv6 and aarch64 to bring them closer to true Tier-1 status.
To further facilitate locating patterns in the Poudriere output, I have begun reworking some existing BROKEN/NOT_FOR/ONLY_FOR messages so that they will sort more easily. This includes sorting the order in which architectures appear in the lists.
Many people have been doing great work on fixing the individual ports. I hope that my work makes their jobs somewhat easier.
Xfce on FreeBSD
| Links | |
| FreeBSD Xfce Project | URL: https://wiki.FreeBSD.org/Xfce |
| Ports Development Repository | URL: https://www.assembla.com/spaces/xfce4/subversion/source |
Contact: FreeBSD Xfce Team <[email protected]>
Contact:
Olivier
Duchateau
<[email protected]>
Xfce is a free software desktop environment for Unix and Unix-like platforms such as FreeBSD. It aims to be fast and lightweight, while still being visually appealing and easy to use.
During this quarter, we have kept these applications up-to-date:
- audio/xfce4-pulseaudio-plugin (0.2.5, PR219357)
- deskutils/xfce4-tumbler (0.1.32, PR219848)
- deskutils/xfce4-xkb-plugin (0.8.0, PR220071)
- sysutils/garcon (0.6.1, PR219928, and PR219334 for Mk/Uses/xfce.mk)
- textproc/xfce4-dict-plugin (0.8.0, PR220266)
- x11/xfce4-terminal (0.8.5.1, PR219312)
- x11/xfce4-whiskermenu-plugin (1.7.2, PR219347)
- x11-wm/xfce4-desktop (4.12.4, PR220290)
We have created a new Subversion tag (4.13) in order to follow the unstable releases. The separate tag was necessary in order to support changes in the USES=xfce infrastucture, and due to some incompatible changes to the xfconf API. Ports following the unstable release are:
- deskutils/xfce4-tumbler (0.1.92.1)
- multimedia/xfce4-parole (0.9.2)
- sysutils/xfce4-settings (4.13.1)
- x11/libexo (0.11.3)
- x11/libxfce4menu (4.13.2)
- x11/libxfce4util (4.13.1)
- x11/xfce4-conf (4.13.2)
- x11/xfce4-dashboard (0.7.2)
- x11/xfce4-screenshooter (1.9.1)
- x11/xfce4-whiskermenu-plugin (2.1.2)
- x11-wm/xfce4-desktop (4.13.1)
- x11-wm/xfce4-panel (4.13.0)
- x11-wm/xfce4-session (4.13.0)
- x11-wm/xfce4-wm (4.13.0)
Open tasks:
- Make the transition to Gtk3 smoother for end users.
Documentation
Absolute FreeBSD, 3rd Edition
Contact: Michael Lucas <[email protected]>
I'm working on a third edition of Absolute FreeBSD. This will be a nearly complete rewrite, thanks to the addition of little details like ZFS, GPT, dma, GELI, new boot procedures, disk labeling, pkg(8), blacklistd, jails, etc..
My current (delusional) plan is to have a first draft finished by the end of October 2017, so we can have print copies for BSDCan 2018.
Open tasks:
- Write the remaining 75% of the book.
Doc Version Strings Improved by Their Absence
Contact: Warren Block <[email protected]>
In retrospect, our $FreeBSD$ strings in source files are kind of weird, like a vestigial tail. The version control system stores all of that information in metadata. Yet here we are, not only allowing the version control system to alter our source files on every commit, but forcing it to do so.
The reason for doing so is that the previous version control system did it. Really.
Version control strings are a headache for translators using the new PO toolchain. It is an ever-changing string that offers nothing to the translation, yet can cause conflicts with earlier versions of itself.
We also had complaints about how the Handbook was always months out of date. It was not, of course... but looking at just the version string in the main, rarely-changing book.xml file gave that impression. We fixed that problem last year, so the build system checks all the source files for the latest commit, but it seems easier to not have to fix the problem at all.
Of course, that was really only one aspect of an ongoing problem. Our documentation build system was checking the version string in the source file, not the metadata. In 1973, metadata, like cars not composed chiefly of rust, had not yet been invented. I modified the build system to extract the information from the metadata (and noted, with some surprise, that this is a task at which Git is much better than Subversion).
The next step was to remove the $FreeBSD$ strings from the source files and remove the FreeBSD=%H property that forces Subversion, against its better judgement, to substitute text in the actual contents of the file. The version information is not lost. It lives in the metadata, so retrieving it is as simple as svn info — it does not need to be in the source at all. However, as with anything that touches code or processes which have not been touched in living memory, there was some debate over this. At that point, I offered to remove the version strings from the FreeBSD Documentation Project Primer book as a test.
The change allowed the zh_TW translation team to turn off the FreeBSD=%H property on their translation and continue their work without fighting with the version strings. Rendered versions of the book still display the name of the last committer and the date and revision number of the last commit, but all of that information comes from metadata. As such, it is also more likely to be correct.
Since the change, there have not been any complaints, at least not to me. In fairness, the removal of version strings from the FDP Primer alone is a small change in a tiny corner of the project. Looking at it another way, it might be that some things that seem to be necessary are more about the comfort of familiarity than actual utility.
At present, this is strictly a change to the documentation build toolchain and a single documentation book. However, there does not appear to be any reason why it could not be extended to the rest of the documents. It might even serve as tiny test of whether the expansion of $FreeBSD$ tags is needed throughout the rest of the FreeBSD tree.
New Xen Handbook Section
Contact: Benedict Reuschling <[email protected]>
FreeBSD supports the Xen hypervisor, with DomU (guest) support since FreeBSD�8.0 and Dom0 (host) available since FreeBSD�11.0. The FreeBSD Handbook was lacking instructions on how to run a Xen host and VMs. The steps were outlined in the FreeBSD wiki, but needed some extra bits of text from the upstream Xen wiki in order to form a complete guide. The new handbook section briefly explains what Xen is, how it differs from other hypervisors, and what features are currently available in FreeBSD. It then goes on to describe how to set up the Dom0, as well as detailing the guest VM support known as DomU.
Reviewers Nikolai Lifanov, Roger Pau Monn�, and Warren Block provided valuable feedback on the initial version in Phabricator. Additional corrections were made by Bj�rn Heidotting while translating the section into German.
Open tasks:
- More options for the Dom0 and DomU could be provided.
- People should test these instructions on their hardware and provide feedback. This would also help us get better testing of the Xen port for FreeBSD.
Miscellaneous
BSD Meetups at Rennes (France)
| Links | |
| First Event | URL: https://www.meetup.com/fr-FR/Meetup-BSD-Rennes/events/239248155/ |
| Second Event | URL: https://www.meetup.com/fr-FR/Meetup-BSD-Rennes/events/240202297/ |
Contact: Mathieu Kerjouan <[email protected]>
Two meetups dedicated to BSD systems were held in Rennes, France. The first one was hosted in the OVH office in Rennes and included presentations on multiple subjects: the non-technical history of FreeNAS (presented by olivier@), how OVH is using ZFS, an introduction to jails, and a use case for BGP/bird on FreeBSD.
The second meetup, also hosted in the OVH office, presented these subjects: how to create a FreeBSD port (presented by jadawin@), how OVH is using Finite State Machines for managing their storage system, network high-availability with FreeBSD, and a jail tutorial by means of a demonstration running 200 OSPF (using net/bird) routers using jails and vnets on a small PC Engines APU2 system with only 4 CPU cores (1Ghz AMD) and 4GB RAM).
This project was sponsored by OVH.
Third-Party Projects
Many projects build upon FreeBSD or incorporate components of FreeBSD into their project. As these projects may be of interest to the broader FreeBSD community, we sometimes include brief updates submitted by these projects in our quarterly report. The FreeBSD project makes no representation as to the accuracy or veracity of any claims in these submissions.
HardenedBSD
Contact:
Shawn
Webb
<[email protected]>
Contact:
Oliver
Pinter
<[email protected]>
HardenedBSD is a derivative of FreeBSD that gives special attention to security-related enhancements and exploit-mitigation technologies. From an initial focus on Address Space Layout Randomization (ASLR), it has now branched out to explore additional exploit mitigation techniques.
It has been a long while since HardenedBSD's last entry in a quarterly status report, back in 2015Q4. The intervening year saw HardenedBSD gain new developers Bernard Spil and Franco Fichtner, import LibreSSL and OpenNTPd into base as the default crypto library and NTP client, respectively, and introduce the hbsd-update binary update mechanism for the base system. The secadm application got a rewrite and Trusted Path Execution (TPE). PIE is now enabled for the base system for arm64 and amd64 as well as the bulk of the ports tree, and the ports tree also gained RELRO and BIND_NOW. Integriforce (similar to NetBSD's verified exec, veriexec) was introduced for the base system, as well as SafeStack, a technology for protection against stack-based buffer overflows that's developed by the Clang/LLVM community. SafeStack relies and builds on top of Address Space Layout Randomization (ASLR), and is strengthened by the presence of PaX NOEXEC. Certain high-profile ports also have SafeStack enabled.
Extremely generous hardware donations from G2, Inc. have provided for dedicated package building and binary update servers, as well as development and test servers.
In March of 2017, we added Control Flow Integrity (CFI) to the base system. CFI is an exploit mitigation technique that helps prevent attackers from modifying the behavior of a program and jumping to undefined or arbitrary memory locations. This type of technique is gaining adoption across the industry — Microsoft has implemented a variant of CFI, which they term Control Flow Guard, or CFG, and the PaX team has spent the last few years perfecting their Reuse Attack Protector, RAP. Of these, RAP is the most complete and effective implementation, followed by Clang's CFI. RAP would be a great addition to HardenedBSD; however, it requires a GPLv3 toolchain and is patent-pending.
CFI can be implemented either on a per-DSO basis, or across all DSOs in a process. Currently only the former is implemented, but we are working hard to enable cross-DSO CFI. As is the case for SafeStack, cross-DSO CFI requires both ASLR and PaX NOEXEC in order to be effective. If an attacker knows the memory layout of an application, the attacker might be able to craft a data-only attack, modifying the CFI control data.
The behavior of several system control (sysctl) nodes has been tighened up, limiting write access and introducing additional safety checks for write accesses. Kernel module APIs received a similar treatment. HardenedBSD's PaX SEGVGUARD implementation received a few updates to make it more stable and performant.
As of March 2017, HardenedBSD is now accessible through a Tor hidden service. The main website, binary updates, and package distribution are all available over the hidden service.
We now maintain our own version of the drm-next branch for updated graphics support. Binary updates are also provided for this branch.
HardenedBSD would like to thank all those who have generously donated time, money, or other resources to the project.
This project was sponsored by SoldierX, and G2, Inc.
Open tasks:
- Port SafeStack to arm64.
- Integrate Cross-DSO CFI.
- Add documentation to the HardenedBSD Handbook.
- Start porting grsecurity's RBAC.
News Home | Status Home