Introduction
As focused as we are on the present and what is happening now, it is sometimes useful to take a fresh look at where we have come from, and where we are going. This quarter, we had our newest doc committer working to trace through the tangled history of many utilities, and we also get a glimpse looking forward at what may come in FreeBSD 12.
Though 11.0-RELEASE was not finalized until after the period covered in this report, we can still have some anticipatory excitement for the features that will be coming in 12.0. The possibilities are tantalizing: a base system with no GPL components, arm64 as a Tier-1 architecture, capsicum protection for common utilities, and the CloudABI for custom software are just a few.
The work of the present is no less exciting, with 11.0 making its way out just after the end of Q3, the new core coming into its own, and much more that you'll have to read and find out.
—Benjamin Kaduk
Please submit status reports for the fourth quarter of 2016 by January 7.
FreeBSD Team Reports
Projects
- Capsicum Update
- ClonOS: New FreeBSD-Based Free/Open Hosting Platform
- CloudABI: Running Untrusted Programs Directly on top of FreeBSD
- Improvements to Non-Transparent Bridge Subsystem
- The Graphics Stack on FreeBSD
- Using lld, the LLVM Linker, to Link FreeBSD
- VirtualBox Shared Folders Filesystem
- ZFS Code Sync with Latest OpenZFS/Illumos
Kernel
- evdev Support
- FreeBSD Driver for the Annapurna Labs ENA
- FreeBSD on Hyper-V and Azure
- Timekeeping Code Improvements
Google Summer of Code
Architectures
Ports
Documentation
FreeBSD Team Reports
FreeBSD Release Engineering Team
| Links | |
| FreeBSD�11.0-RELEASE schedule | URL: https://www.FreeBSD.org/releases/11.0R/schedule.html |
Contact: FreeBSD�Release Engineering Team <[email protected]>
The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes, and maintaining the respective branches, among other things.
The FreeBSD Release Engineering Team continued the 11.0-RELEASE cycle which was planned to be released in September, but as a result of several last-minute issues, the final release announcement was delayed.
This project was sponsored by The FreeBSD Foundation.
Ports Collection
Contact:
Ren�
Ladan
<[email protected]>
Contact: FreeBSD Ports Management Team <[email protected]>
The Ports Tree currently contains over 26,300 ports, with the PR count around 2,150. Of these PRs, 516 are unassigned. The last quarter saw 5,295 commits by 117 active committers. Compared to the preceding quarter, there is both a slight increase in the number of PRs and the number of unassigned PRs, and a slight decrease in the number of committers.
In the last quarter, four commits bits were taken in for safe keeping: erwin, miwi, and sem left by their own request and jase was inactive for more than 18 months. We welcomed two new committers: Tobias Berner (tcberner) and Joseph Mingrone (jrm).
On the management side, erwin and miwi left portmgr. bapt also left portmgr but is still the liaison for core.
On the infrastructure side, three new USES (grantlee, kde, linux) and one new Keyword (javavm) were introduced. The default version of the Linux ports is now CentOS 6, with the Fedora 10 ports scheduled for removal at the end of the year. The license framework has been extended with a NONE license to indicate that a port has no clearly defined licensing terms. For those ports, no packages or distribution files are distributed. Also, support for the complete set of Creative Commons licenses has been added.
Some major user-visible ports were updated: Firefox to 49.0 and Firefox Extended Service Release to 45.4.0; Chromium to 52.0.2743.116; the default version of gcc to 4.8.5; and pkg itself to 1.8.7.
Behind the scenes, antoine ran 24 exp-runs to validate various package updates, framework changes, and changes to the base system. bdrewery added two new package building machines, supervised the package builds for 11.0-RELEASE, and added support for building arm64 packages.
At EuroBSDcon, rene visited a presentation by Landry Breuil <[email protected]> explaining how packages are built in the OpenBSD world and explaining various design decisions.
Open tasks:
- If you have some spare time, please take up a PR for testing and committing.
The FreeBSD Core Team
Contact: FreeBSD Core Team <[email protected]>
The third quarter started with the handover to the ninth Core team as it took office. With four members returning from the previous core (Baptiste Daroussin, Ed Maste, George Neville-Neil and Hiroki Sato), one returning member after a term away (John Baldwin), and four members new to core (Allan Jude, Kris Moore, Benedict Reuschling and Benno Rice), the new core team represents just about the ideal balance between experience and fresh blood.
Beyond handing over all of the ongoing business, reviewing everything on Core's agenda, and other routine changeover activities, the first action of the new core was to respond to a query from Craig Rodrigues concerning how hardware supplied to the project through donations to the FreeBSD Foundation was being used.
The Foundation does keep records of what hardware has been supplied over time and has some idea of the original purpose that hardware was provisioned for, but does not track the current usage of the project's hardware assets. Cluster administration keeps their own configuration database, but this is not suitable for general publication and covers much more than Foundation supplied equipment. After some discussion it was decided that updated information about the current disposition of Foundation supplied equipment should be incorporated in the Foundation's annual report.
Ensuring that all of the FreeBSD code base is supplied under open and unencumbered licensing terms and that we do not infringe on patent terms or otherwise act counter to any legal requirements are some of Core's primary concerns. During this quarter, there were three items of this nature.
- Importing Concurrency Kit. In consultation with the Foundation's legal counsel, it was determined that importing selected parts of concurrency kit is acceptable, and has been approved.
- The proposal to create a shadow GPLv3 toolchain repository was put to the community. Ultimately the whole idea has been rendered largely redundant by faster than anticipated progress on the external toolchain ports and packages for those architectures where LLVM is not yet sufficiently mature.
- Concerns were raised about handling GPL code in work in progress on the linuxkpi shim. This issue is not related to the FreeBSD svn repository but Core would like to stress that care must be taken to avoid license infringement and plans to write a set of guidelines for handling GPL code.
The item that has absorbed the largest portion of Core's attention this quarter concerns the project's handling of security vulnerabilities in bspatch(1), libarchive(3), freebsd-update(8) and portsnap(8). A partial fix was applied in FreeBSD-SA-16:25.bspatch but this lacks fixes to libarchive code that were not yet available from upstream.
SecTeam receives privileged early reports of many vulnerabilities and consequently has a strict policy of not commenting publicly until an advisory and patches have been published. Early access to information about vulnerabilities is contingent on their ability to avoid premature disclosure, and without such, they could not have security advisories and patches ready to go immediately when a vulnerability is published.
However, in this case, vulnerabilities were already public and the lack of any official response from the FreeBSD Project was leading to concern amongst users and some critical press coverage. Core stepped in and published a statement clarifying the situation and the particular difficulties involved in securely modifying the mechanisms used to deliver security patches. Core believes that prompt notification and discussion of the implications and possible workarounds to any public vulnerability should not wait on the availability of formal OS patches.
The OpenSSH project has deprecated DSA keys upstream. FreeBSD had kept DSA keys enabled in the later 10.x releases for compatibility reasons, but with the release of 11.0 the time has come to synchronize again with upstream. Since there are numerous DSA keys in use in the FreeBSD cluster, this necessitated an exercise to get replacement keys installed. Core would like to thank David Wolfskill and the accounts team for handling the surge in key changes with a great deal of aplomb.
During this quarter we welcomed Michael Zhilin, Imre Vadasz, Steve Kiernan and Toomas Soome as new source committers. Over the same period, we said farewell to Martin Wilke and Erwin Lansing who have handed in their commit bits. We wish them well in their future endeavours and hope to see them return as soon as they can.
The FreeBSD Foundation
| Links | |
| FreeBSD Foundation Website | URL: https://www.FreeBSDFoundation.org/ |
Contact: Deb Goodkin <[email protected]>
The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases hardware to improve and maintain FreeBSD infrastructure and publishes FreeBSD white papers and marketing material to promote, educate, and advocate for the FreeBSD Project. The Foundation also represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity.
Here are some highlights of what we did to help FreeBSD last quarter:
Fundraising Efforts
Our work is 100% funded by your donations. Our spending budget for 2016 is $1,250,000 and we have raised $271,500 so far. Our Q1-Q3 financial reports will be posted in early November. As you can see, we need your donations to continue supporting FreeBSD at our current level. Please consider making a donation at https://www.FreeBSDFoundation.org/donate/.
OS Improvements
The Foundation improves FreeBSD by funding software development projects approved through our proposal submission process, and our internal software developer staff members. Two Foundation-funded projects continued last quarter: one project is to port NetBSD's blacklistd daemon and related elements to FreeBSD, and the second is phase two of the FreeBSD/arm64 port.
Foundation staff members were responsible for many changes over the quarter. Kostik Belousov accomplished this work last quarter:
- Provided kernel support for EFI Runtime Services calls
- Implemented gettimeofday(2) purely in userspace for HPET timers
- Implemented fdatasync(2)
- Improved the locking of the time keeping code
- Made the sleepqueue code immune to rapid callout changes
- Made many stability fixes, the most important of which were UFS issues and an i386 bug
- Improved the process management and ptrace code
Ed Maste, our Project Development Director, accomplished this work last quarter:
- Worked on FreeBSD/arm64 issues and Cavium ThunderX deployment (including RMAs)
- Worked with upstream developers to test works in progress and prepare LLD as the replacement linker in the FreeBSD base system
- Switched to using LLVM's libunwind in the base system
- Improved the reproducibility of builds in the FreeBSD base system and ports
- Reviewed the blacklistd work that is in progress
- Attended BSDCam 2016, with a primary focus on toolchain discussions
- Participated in ongoing Capsicum calls, and helped with the Capsicumization of several base system utilities
- Fixed a number of ELF Tool Chain issues, and integrated a new upstream version into the FreeBSD base system
- Hosted biweekly graphics calls to coordinate work in progress by funded and volunteer developers
- Implemented fixes for security issues in some FreeBSD update tools, and coordinated their integration into the stable and release branches
George Neville-Neil continued hosting a bi-weekly Transport conference call (notes at https://wiki.FreeBSD.org/TransportProtocols) and the bi-weekly DTrace conference call (notes at https://wiki.FreeBSD.org/DTrace).
Release Engineering
Foundation staff member Glen Barber worked with the Release Engineering team to continue finalizing the 11.0-RELEASE cycle, which was delayed to address several last-minute issues.
As part of the Cluster Administration team, Glen worked with the amazing on-site staff at NYI to rack and install two Cavium ThunderX machines, one of which is used for native package builds for the FreeBSD/arm64 architecture, and the other of which is targeted to be used as a reference machine in the FreeBSD infrastructure.
Getting Started with FreeBSD Project
We hired a summer intern, with no experience on FreeBSD, Linux, or any command-line operating system, to figure out on his own how to install and use FreeBSD. He wrote easy-to-follow how-to guides to help make the new-user experience straightforward and positive. He submitted bug reports and reported issues through the appropriate channels, and worked with Glen Barber and Brad Davis to improve the new user information on FreeBSD.org to make it easier for new people to get started with FreeBSD. You can find his how-to guides at https://www.FreeBSDFoundation.org/FreeBSD/how-to-guides/ and check out his interview on BSDNow at http://www.bsdnow.tv/episodes/2016_08_24-the_fresh_bsd_experience.
Supporting FreeBSD Infrastructure
We provide hardware and support for FreeBSD infrastructure. Last quarter we purchased and brought up two 48-core Cavium ThunderX machines to build FreeBSD package sets for the arm64 platform. We also purchased more servers to help with continuous integration efforts.
FreeBSD Advocacy and Education
A large part of our efforts are dedicated to advocating for the Project. This includes promoting work being done by others using FreeBSD, producing advocacy literature to teach people about FreeBSD and ease the path to starting out with FreeBSD, contributing to the Project, and attending and getting other FreeBSD contributors to volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD presentations.
We created new handouts to promote TeachBSD.org (https://www.FreeBSDFoundation.org/wp-content/uploads/2016/08/TeachBSD_half_final.pdf) and the Google Summer of Code program (https://www.FreeBSDFoundation.org/wp-content/uploads/2016/08/GSOC-flyerv2.pdf).
We published the July/August issue of the FreeBSD Journal: https://www.FreeBSDFoundation.org/past-issues/FreeBSD-and-rtems/.
We also published monthly newsletters to highlight work being done to support FreeBSD, tell you about upcoming events, and provide other information to keep you in the loop of what we are doing to support the FreeBSD Project and community: https://www.FreeBSDFoundation.org/news-and-events/newsletter/.
Conferences and Events
The FreeBSD Foundation sponsors many conferences, events, and summits around the globe. These events can be BSD-related, open source, or technology events geared towards underrepresented groups.
We support the FreeBSD-focused events to help provide a venue for sharing knowledge, to work together on projects, and facilitate collaboration between developers and commercial users. This all helps provide a healthy ecosystem. We support the non-FreeBSD events to promote and raise awareness about FreeBSD, to increase the use of FreeBSD in different applications, and to recruit more contributors to the Project.
This quarter, we sponsored and/or attended the following events:
- Texas Linux Fest, July 8-9, 2016, Austin, TX (http://2016.texaslinuxfest.org/)
- The Eleventh HOPE, July 22-24, 2016, New York, NY (https://hope.net/index.html)
- BSDCam 2016, August 15-17, 2016, Cambridge, UK (sponsor, organizer, and participated) (https://wiki.FreeBSD.org/DevSummit/201608)
- FOSSCON 2016, August 20, 2016, Philadelphia, PA (https://fosscon.us/)
- womENcourage 2016, September 12-13, 2016, Linz, Austria (Silver Sponsor) (http://womencourage.acm.org)
- SNIA Storage Developer Conference 2016, September 19-22, 2016, Santa Clara, CA (Industry Partner Sponsor) (http://www.snia.org/events/storage-developer)
EuroBSDcon 2016 and FreeBSD Developer Summit, September 22-25, 2016, Belgrade, Serbia (Silver Sponsor) (https://2016.eurobsdcon.org/)
Our EuroBSDcon involvement included:
- Held a Women in Tech BoF in partnership with ACM-W Europe
- Benedict organized the EuroBSDcon Developer Summit
- Deb gave a Foundation Update talk and Hiroki Sato and Benedict Reuschling joined her for a Q&A session.
- Kirk McKusick taught his two-day FreeBSD tutorial (https://2016.eurobsdcon.org/speakers/#kirkmckusick)
- George Neville-Neil taught a tutorial on Tracing FreeBSD for DevOps and Developers (https://2016.eurobsdcon.org/speakers/#georgenevilleneil)
- George also gave the Keynote talk, titled The Coming Decades of BSD
- Phillip Paeps was one of the primary organizers for this conference.
OpenZFS Developer Summit 2016, September 26-27, 2016, San Francisco, CA (Silver) (http://open-zfs.org/wiki/OpenZFS_Developer_Summit)
- Justin Gibbs gave a talk on Fault Management (http://open-zfs.org/wiki/Fault_Management)
We sponsored three FreeBSD contributors to attend EuroBSDcon.
Legal/FreeBSD IP
The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We continued to review requests and grant permission to use the trademarks.
FreeBSD Community Engagement
Anne Dickison, our Marketing Director, has been overseeing the efforts to rewrite the Project's Code of Conduct to help make this a safe, inclusive, and welcoming community.
Other Stuff We Did
We welcomed Kylie Liang and Philip Paeps to the Board of Directors. More information and interviews can be found at: https://www.FreeBSDFoundation.org/blog/FreeBSD-foundation-welcomes-new-board-members/.
George attended the ARM Partner Meeting in Cambridge.
Projects
Capsicum Update
| Links | |
| Capsicum Wiki Page | URL: https://wiki.FreeBSD.org/Capsicum |
Contact:
Allan
Jude
<[email protected]>
Contact:
Baptiste
Daroussin
<[email protected]>
Contact:
Conrad
Meyer
<[email protected]>
Contact:
Ed
Maste
<[email protected]>
Contact:
Mariusz
Zaborski
<[email protected]>
Several developers have undertaken a recent effort to sandbox additional applications in the base system. This work is proceeding nicely and one of the goals is to target basic utilities used in security sensitive applications, like freebsd-update and portsnap.
This work highlighted two longstanding challenges in applying Capsicum. First, there are a number of common constructs shared by many simple programs, such as limiting capability rights on the stdio file descriptors. To address this, a set of capsicum helper routines has been added for these common cases.
Second, a common challenge occurs where applications need to open an arbitrarily large number of files, possibly from various directories, where preopening the file descriptors may not be suitable. Several possible solutions for this are in discussion.
Recently Capsicumized utilities include:
- bspatch
- cmp
- ident
- primes
- tee
- tr
- write
Additional Capsicum changes are in review:
- b64decode, b64encode, uudecode, uuencode
- brandelf
- dma-mbox-create
- elf2aout
- file
- head
- hexdump
- iconv
- ident
- jot
- ktrdump
- lam
- last
- ministat
- praudit
- strings
An additional syscall (getdtablesize) and additional sysctls (kern.proc.nfds, kern.hostname, etc.) are now permitted in capability mode.
Capability rights are now propagated to child descriptors on accept(2).
Capsicum is now enabled in the 32-bit compatibility syscall layer.
Per-process (procctl) and global (sysctl) settings have been added to aid in debugging while Capsicumizing existing applications. When enabled, instead of returning ENOTCAPABLE or ECAPMODE for a system call, the kernel will issue a SIGTRAP to generate a core dump or enter the debugger.
This project was sponsored by Dell EMC Isilon, ScaleEngine Inc., and The FreeBSD Foundation.
ClonOS: New FreeBSD-Based Free/Open Hosting Platform
| Links | |
| ClonOS Homepage | URL: http://clonos.tekroutine.com |
Contact: Oleg Ginzburg <[email protected]>
Currently, FreeBSD is well proven as a base for routers (pfSense, OPNSense, BSDRP) and NAS (FreeNAS, zfsGuru, NAS4Free). However, FreeBSD-based solutions are almost completely absent in the virtualization area, and ClonOS is one of the attempts to change that.
ClonOS is a new free open-source FreeBSD-based platform for virtual environment creation and management. In the core platform are:
- FreeBSD as the host OS
- bhyve
- xen
- vale
- jail
- CBSD (as a management tool)
- puppet (for configuration management)
- additional features such as go-micro services (obtaining VMs, resizing disks, and so on)
Open tasks:
- We would like to see ClonOS in real-world use. In this regard we are interested in finding more people and companies that use FreeBSD in hosting tasks. In addition, it could be great to work with the developers of existing NAS solutions (zfsGuru, NAS4Free).
CloudABI: Running Untrusted Programs Directly on top of FreeBSD
Contact:
Ed
Schouten
<[email protected]>
Contact: The CloudABI mailing list <[email protected]>
CloudABI is a compact UNIX-like runtime environment inspired by FreeBSD's Capsicum security framework. It allows you to safely run potentially untrusted programs directly on top of FreeBSD, Linux and macOS, without requiring the use of virtualisation, jails, etc. This makes it a useful building block for cluster/cloud computing.
Over the last couple of months, several new libraries and applications have been ported over to CloudABI, the most important addition being Python 3.6. This means that you can now write strongly sandboxed apps in Python!
Support for different hardware platforms has also improved. In addition to amd64 and arm64, we now support i686 and armv6. The release of LLVM 3.9 was important to us, as it has integrated all the necessary changes to support the first three platforms. Full armv6 support is still blocked on some issues with LLVM's linker, LLD.
This project was sponsored by Nuxi, the Netherlands.
Open tasks:
- Play around with CloudABI and let us know what you think of it! Full support for amd64 and arm64 is part of FreeBSD 11.0. i686 and armv6 support is only available on head, but will be merged to the stable/11 branch in the future.
- Interested in Python programming? Give our copy of Python a try and share your experiences!
- Do you maintain pieces of software that could benefit from strong sandboxing? Try building them using the CloudABI cross compiler!
Improvements to Non-Transparent Bridge Subsystem
Contact: Alexander Motin <[email protected]>
Non-Transparent Bridges allow the creation of memory windows between different systems using the regular PCIe links of CPUs as a transport. During the last quarter, the NTB subsystem gained a significant set of improvements and fixes:
- The code was modularized, utilizing FreeBSD's NewBus interfaces to allow support for different hardware types with different drivers, support for multiple NTB instances in a system, using the ntb_transport module for consumers other than if_ntb, etc.
- Support for splitting NTB resources between different applications was added, such as doing direct access to some range of remote memory and to a virtual network interface between nodes at the same time, etc.
- The virtual network interface driver gained support for many modern features, such as multiple queues, new locking, etc.
- NTB performance and SMP scalability was improved.
- Multiple workarounds for hardware issues were added.
The code is committed to the FreeBSD head, stable/11 and stable/10 branches.
This project was sponsored by iXsystems, Inc.
Open tasks:
- Support for the next generation of Intel hardware.
- Support for non-Intel hardware (AMD, PLX, etc.).
- Support for I/OAT and other DMA offloads.
- Creating a more efficient packet transport protocol.
- Creating a greater variety of NTB applications.
The Graphics Stack on FreeBSD
Contact: FreeBSD Graphics Team <[email protected]>
Contact:
Matthew
Macy
<[email protected]>
Contact:
Johannes
Lundberg
<[email protected]>
We are sad to report that both GSoc projects failed. The libdevq project was abandoned as the student disappeared. The kernel project was incomplete because the student could not work for personal reasons. He plans to resume work and complete the task, even though GSoC 2016 is finished.
X.org server version 1.18.4 and updates for the xf86-video-ati and xf86-video-intel DDX drivers are ready for wider testing. A CFT will be sent out shortly. These updates are required to use newer DRM versions.
The missing functionality from libdrm that is needed by the amdgpu driver has been added. These changes will be committed to the ports tree shortly after the xorg-server update.
DRM from Linux 4.8 was ported to the drm-next branch. This branch should be used for radeon and amdgpu cards. The drm-next-4.7 branch should be used for i915 cards due to instabilities in the intel driver in the drm-next branch.
Johannes Lundberg has been working on getting the Wayland environment running on FreeBSD. The Wayland ports are in a working state except for the Weston compositor.
The current Weston port (from DragonFlyBSD) might be scrapped and a new port created from scratch based on the upstream source code. With the use of libinput, libudev-devd, and epoll-shim, the diff will not be very large and will be easier to maintain.
Patches for wlc (another Wayland compositor) are being pushed upstream. On the TODO list is refactoring the tty code into selectable backends (linux, FreeBSD, etc), as recommended by the author of wlc. For now, it is running on FreeBSD with patches in the ports tree.
Using lld, the LLVM Linker, to Link FreeBSD
| Links | |
| LLD Wiki Page | URL: https://wiki.FreeBSD.org/LLD |
Contact: Ed Maste <[email protected]>
lld is the linker in the LLVM family of projects. It is a high-performance linker that supports the ELF, COFF, and Mach-O object formats. Where possible, lld maintains command-line and functional compatibility with the existing GNU BFD ld and gold linkers. However, the authors of lld are not constrained by strict compatibility where it would hamper performance or desired functionality.
Compared to the GNU ld 2.17.50 currently in the base system, lld will bring:
- AArch64 (arm64) support
- Link Time Optimization (LTO)
- New ABI support
- Other linker optimizations
- Much faster link times
- Maintained code base
The upstream lld project has now implemented nearly all of the functionality required to link the amd64 FreeBSD base system, including the kernel. The boot loader components and rescue utilities currently do not build with lld.
This project was sponsored by The FreeBSD Foundation.
Open tasks:
- Merge lld to FreeBSD head as part of the Clang 3.9.0 import.
- Request a ports exp-run with lld installed as /usr/bin/ld.
- Fix building the boot loader with lld.
- Fix building rescue with lld.
- Test and iterate making lld fixes for additional architectures.
VirtualBox Shared Folders Filesystem
| Links | |
| Project Repository | URL: https://github.com/lwhsu/FreeBSD-vboxfs |
Contact:
Li-Wen
Hsu
<[email protected]>
Contact:
Oleksandr
Tymoshenko
<[email protected]>
FreeBSD provides an API for guest operating systems to access shared folders on the host so that the kernel driver can expose them to the guest's userland. This project aims to add such functionality to the VirtualBox Guest Additions driver.
Good progress was made over the last few months. Developers were able to mount a filesystem in read-only mode and, with some limitations, in read-write mode. The implementation still lacks some critical pieces, but the roadmap is clear.
Open tasks:
- Finish the missing pieces.
- Implement proper locking.
- General clean-up and bugfixes.
ZFS Code Sync with Latest OpenZFS/Illumos
Contact:
Alexander
Motin
<[email protected]>
Contact:
Andriy
Gapon
<[email protected]>
The ZFS code base in FreeBSD regularly gets merges of new code, staying in sync with the latest OpenZFS/Illumos sources. Among other things, the latest merge included the following improvements:
- The ARC now mostly stores compressed data, the same as is stored on disk, decompressing them on demand.
- The L2ARC now stores the same (compressed) data as the ARC without recompression, and its RAM usage was further reduced.
- The largest size of indirect block possible has been increased from 16KB to 128KB, and speculative prefetching of indirect blocks is now performed.
- Improved ordering of space allocation.
- The SHA-512t256 and Skein hashing algorithms are now supported.
Kernel
evdev Support
| Links | |
| evdev WIP Repository | URL: https://github.com/wulf7/FreeBSD |
| Original evdev Proposal | URL: https://wiki.FreeBSD.org/SummerOfCode2014/evdev_Touchscreens |
Contact:
Vladimir
Kondratiev
<[email protected]>
Contact:
Oleksandr
Tymoshenko
<[email protected]>
evdev is a portable, API-compatible implementation of the Linux /dev/input/eventX interface. It covers a wide variety of input devices like keyboards, mice, and touchscreens (with multitouch support), and support for it is implemented in a lot of existing userland components like Qt, libinput, and tslib.
evdev support was started by Jakub Klama as a Google SoC 2014 project, and later picked up and finished by Vladimir Kondratiev. General API and evdev support bits for ukbd and ums were committed to head. Support was also added for TI's AM33xx touchstreen controller (the popular BeagleBone is based on the AM33xx) and the official touchscreen for the Raspberry Pi. Multitouch support for the Raspberry Pi was successfully demonstrated using the latest Qt development branch.
Open tasks:
- Documentation. In particular, manual pages are needed for the KPI.
- Support additional hardware.
- Enable evdev support in existing ports, and add new evdev-dependent ports.
FreeBSD Driver for the Annapurna Labs ENA
| Links | |
| Amazon AWS Documentation of the ENA | URL: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html |
Contact:
Jan
Medala
<[email protected]>
Contact:
Jakub
Palider
<[email protected]>
The Elastic Network Adapter (ENA) is a 25G SmartNIC developed by Annapurna Labs based on a custom ARMv8 chip. This is a high-performance networking card that is available to AWS virtual machines. It introduces enhancements in network utilization scalability on EC2 machines running various operating systems, in particular FreeBSD.
The goal of FreeBSD enablement is to provide top performance and a wide range of monitoring and management features such as:
- multiple queue modes
- various offload functionality
- admin queue
- asynchronous notification
- robust hardware access
- scalable number of MSI-X interrupts
- counters
The current state offers stable driver operation with good performance on machines running FreeBSD directly on the hardware.
This project was sponsored by Annapurna Labs — an Amazon company.
Open tasks:
- Optimize performance for virtualized environments.
- Prepare for submitting the driver as a Phabricator review.
FreeBSD on Hyper-V and Azure
Contact:
Sepherosa
Ziehau
<[email protected]>
Contact:
Hongjiang
Zhang
<[email protected]>
Contact:
Dexuan
Cui
<[email protected]>
Contact:
Kylie
Liang
<[email protected]>
This quarter, the Hyper-V storage driver was greatly improved: its performance was increased by a factor of 1.2-2 by applying BUS_DMA and UNMAP_IO, enlarging the request queue, and selecting the outgoing channel with the LUN considered; TRIM/UNMAP was enabled; and some critical bugs (PRs 209443, 211000, 212998) were fixed so that disk hot add/remove and VHDX online resizing should work now.
The VMBus driver also received attention, with enhancements made for the handling of device hot add/remove.
In the Hyper-V network driver, configurable RSS key and dynamic MTU change are now supported.
FreeBSD images on Azure continue to be updated — after publishing the FreeBSD 10.3 VM image on the global Microsoft Azure in June, Microsoft also published the VM image on the Microsoft Azure operated by 21Vianet in China in September.
Patches have been developed to support PCIe pass-through (also known as Discrete Device Assignment); this feature allows physical PCIe devices to be passed through to FreeBSD VMs running on Hyper-V (Windows Server 2016), giving them near-native performance with low CPU utilization. The patch to enable the feature will be posted for review soon.
This project was sponsored by Microsoft.
Timekeeping Code Improvements
Contact: Konstantin Belousov <[email protected]>
Work was done to properly lock the time-keeping code. The existing code correctly interoperated with readers, both kernel- and user-space, giving them lock-less access to the actual data ('timehands') needed to derive the time of day from the timecounter hardware in the presence of updaters. But updates of the timehands, which are performed by periodic clock interrupts, the ntpd-driven sys_ntp_adjtime(2) syscall, the settimeofday(2) syscall, pps sync, and possibly other sources, were not coordinated. Moreso, the NTP code was locked by Giant, which did not really serve any purpose.
As a result of the work, locking was applied to ensure that any timehands adjustments are performed by a single mutator. An interesting case is the parallel modification of the timehands from the top of the kernel, for instance the settimeoday(2) syscall, and a simultaneous clock tick event, where the syscall has already acquired the resources. In this case, it is highly desirable to not block (spin) in the tick handler, and the required adjustments are performed by the already executing call from the top half. There, the typical trylock operation is desired, which was surprisingly lacking in our spinlock implementation. mtx_trylock_spin(9) was implemented and is used for this purpose.
The userspace gettimeofday(2) implementation was enhanced to allow syscall-less operation on machines that use HPET hardware for timecounters. The HPET algorithm coexists with older RDTSC-based code, allowing dynamic switching of timecounters. A page with HPET registers is mmap(2)-ed readonly by libc into userland application programs' address space as needed. Measurements demonstrated modest improvements in gettimeofday(2) performance, but, not unexpectedly, even the syscall-less HPET timecounter is slower than invoking a syscall for RDTSC.
Some not strictly intertwined but related code is the time-bound sleep implementation. Handling of races between callouts and the top-half code that sets and processes the timeouts depended on the many fine details of the callout_stop(9) KPI (kernel programming interface). In particular, races or unpunctual KPI changes usually result in the "catch-all" unkillable thread state with the "-" waitchain bug. The sleepqueue timeout code was rewritten to stop depending on the KPI details, which removed the source of recurring bugs, and also surprisingly simplified the code.
This project was sponsored by The FreeBSD Foundation.
Google Summer of Code
Google Summer of Code 2016
| Links | |
| GSoC 2016 Projects | URL: https://wiki.FreeBSD.org/SummerOfCode2016Projects |
| GSoC Ideas page | URL: https://wiki.FreeBSD.org/SummerOfCodeIdeas |
Contact:
Gavin
Atkinson
<[email protected]>
Contact:
Pedro
Giffuni
<[email protected]>
As in all previous editions of the Google Summer of Code, FreeBSD was an accepted organization, and we had the chance to mentor 15 projects. Huge thanks to all our mentors for keeping the high quality standards that make our community shine.
This year was rather unique in that we accepted for the first time well-known members of the community that are not src committers to co-mentor. We also accepted projects that have a different upstream than FreeBSD. Both are clear signs that FreeBSD is growing and adapting to the wider community.
This year we are also had administrative issues with Perforce and have officially accepted the use of external repositories, in particular github, as requested by students.
12 of 15 projects were successful, which we think is an excellent result for a Google Summer of Code.
This project was sponsored by Google Inc, and The FreeBSD Foundation.
Open tasks:
- The world is changing and we need fresh project ideas. We need to start looking for those ideas now.
- The project ideas wiki page has been reset and we need to get it populated before applying for the next Google Summer of Code. Please help unleash the next stream of projects you want to see in FreeBSD.
Non-BSM to BSM Conversion Tools
Contact: Mateusz Piotrowski <[email protected]>
This project was started during Google Summer of Code this year. The aim was to create a library which can convert the audit trail files in Linux Audit format or the format used by Windows to the BSM format used by FreeBSD for its audit logs. Apart from that, I wanted to create a simple command-line tool and extend auditdistd so that it is possible to send non-BSM logs to it over a secure connection and save those audit logs on disk, preferably in the BSM format.
So far, it is possible to reasonably convert some of the most common Linux audit log events to BSM, but it still needs a lot of work. Secondly, I was able to configure auditdistd to communicate with CentOS over an insecure connection. Thirdly, the command-line tool is usable but not perfect.
The present work focuses on configuring the secure TLS connection between CentOS and auditdistd. I have already tried using rsyslogd but was not able to make it work.
This project was sponsored by Google Summer of Code.
Open tasks:
- I need more examples of rare Linux Audit logs; please send me some examples if you have any. It is much easier to improve the conversion process with real-life examples of audit events as you write the code to convert them.
- Configure auditdistd to be able to communicate with some software on CentOS over TLS in order to receive audit logs. I was not able to come up with a simple solution for that.
- Additional open tasks are listed on the Wiki page and in the TODO file in the root directory of the project.
ptnet Driver and bhyve Device Model
Contact: Vincenzo Maffione <[email protected]>
This project provides:
- A new driver (if_ptnet) for a paravirtualized network device, modeled after the netmap API. The driver supports multi-queue netmap ports, and it is able to work both in netmap mode and in normal mode.
- The emulation of the ptnet device model as a module of the bhyve hypervisor.
The ptnet device and driver has been introduced to overcome the performance limitations of TCP/IP networking between bhyve VMs. Prior to this work, the most performant solution for VM-to-VM intra-host TCP communication provided less than 2 Gbps TCP throughput. With ptnet, in the same VM-to-VM TCP communication scenario, it is possible to obtain up to 20 Gbps.
This project was sponsored by Google Summer of Code.
Open tasks:
- Share virtio-net header management code with the if_vtnet driver. In the current code, about 100 lines of code have been copied and pasted from if_vtnet.c.
Architectures
FreeBSD on Annapurna Labs Alpine
Contact:
Jan
Medala
<[email protected]>
Contact:
Michal
Stanek
<[email protected]>
Contact:
Wojciech
Macek
<[email protected]>
Alpine is a family of Platform-on-Chip devices, including multi-core 32-bit (first-gen Alpine) and 64-bit (Alpine V2) ARM CPUs, developed by Annapurna Labs.
The primary focus areas of the Alpine platform are high-performance networking, storage, and embedded applications. The network subsystem features 10-, 25-, and 50-Gbit Ethernet controllers with support for virtualization, load-balancing, hardware offload and other advanced features.
A basic patch set has already been committed to head including:
- PCIe Root Complex support
- Cache Coherency Unit driver
- North Bridge Service driver
- Updated Alpine HAL
- Extended MSI support in GICv2 and GICv3 code
Additional work, such as an MSI-X driver and full Ethernet support, is currently undergoing community review on Phabricator.
The multi-user SMP system is stable and fully working, along with the 1G and 10G Ethernet links.
The interrupt management code has been adjusted to work with the new INTRNG framework on both ARM32 and ARM64.
This project was sponsored by Annapurna Labs — an Amazon company, and Semihalf.
FreeBSD on Marvell Armada38x
Contact:
Marcin
Wojtas
<[email protected]>
Contact:
Bartosz
Szczepanek
<[email protected]>
FreeBSD includes support for the Marvell Armada38x platform, which has been tested and improved in order to gain production quality. Most of this effort has been invested in development and benchmarking of the on-chip Gigabit Ethernet (NETA) functionality. Numerous bug fixes and some new features have been introduced.
Work completed this quarter includes:
- NETA rework and improvements.
- Enable multi-port support in PCIe 2.0 driver (mv_pci_ctrl).
- Introduce an alternative, coherent, bus_dma for the armv7 arch.
- AHCI controller support.
- SDHCI controller support.
- Improve the e6000sw etherswitch driver.
- Fix Marvell bus configuration for numerous interfaces.
Along with support for new boards (SolidRun ClearFog and DB-88F6285-AP), all changes will be submitted upstream.
This project was sponsored by Stormshield, and Semihalf.
Open tasks:
- Finalize NETA and prepare for submission.
- Submit remaining fixes and drivers.
FreeBSD/arm64
| Links | |
| FreeBSD arm64 Wiki Entry | URL: https://wiki.FreeBSD.org/arm64 |
| Using Crochet to Build FreeBSD Images | URL: https://github.com/wca/crochet/tree/add-pine64-support |
Contact:
Andrew
Turner
<[email protected]>
Contact:
Jared
McNeill
<[email protected]>
Transparent superpage support has been added. This allows FreeBSD to create 2MiB blocks with a single pagetable and TLB entry. This shows a small but significant improvement in the buildworld time on ThunderX machines. Superpages have been enabled in head and merged to stable/11, but they are disabled by default on stable/11 due to a lack of testing there.
Support for the pre-INTRNG interrupt framework has been removed. This means that arm64 requires INTRNG to even build. This has allowed various cleanups within the arm64 drivers that interact with the interrupt controller.
The cortex Strings library from Linaro has been imported. The parts of this that have been shown to be improvements over the previous C code were attached to the libc build.
There is ongoing work to add ACPI support to the kernel. On ThunderX, FreeBSD can get to the mountroot prompt, however, due to incomplete ACPI tables the external PCIe support needed to support the netboot setup in the test cluster is not functional.
Pine64 support has been committed to head. FreeBSD can now boot to multiuser with SMP enabled. This includes support for clocks, secure ID controller, USB Host controller, GPIOs, non-maskable interrupts, AXP81x power management unit, cpu freqency and voltage scaling, MMC, UART, gigabit networking, watchdog, and thermal sensors.
This project was sponsored by The FreeBSD Foundation, and ABT Systems Ltd.
UEFI Runtime Services
Contact: Konstantin Belousov <[email protected]>
UEFI (Unified Extensible Firmware Interface) specifies two kinds of services for use by operating systems. Boot Services are designed for OS loaders to load and initialize kernels, while Runtime Services are meant to be used by kernels during regular system operations. The boot and runtime phases are explicitly separated. During boot, when loaders are executed, the machine configuration is owned by UEFI. During runtime, the kernel manages the configuration, but needs to inform the firmware about any changes that are made.
The model of split boot/runtime configuration makes assumptions about the OS architecture that do not quite apply to the existing FreeBSD codebase. For instance, the firmware notification of the future runtime configuration must be done while the loader is effectively still in control. In technical terms, the SetVirtualAddressMap() call must be made with the 1:1 physical:virtual mapping on amd64 systems, which for FreeBSD means that the call can only be issued by the loader. But the loader needs to know intimate details of the kernel address map to provide the requested information. This creates a new, unfortunate, coupling between loader and kernel.
Reading the publicly available information about the MS Windows boot process explained the UEFI control transfer model. The Windows loader constructs the address map for the kernel, and with such a division of work the UEFI model is reasonable. The FreeBSD kernel constructs its own address space, only relying on a minimal map constructed by the loader, which is enough for the pmap subsystem to bootstrap itself and then to perform machine initialization in common code.
Initial experiments with enabling runtime services were centered around utilizing the direct address map (DMAP) on amd64, which currently always exists and linearly maps at least the lower 4G of physical addresses at some KVA location. It was supposed that the kernel would export the DMAP details like linear base and guaranteed size for loader from its ELF image, and provide the needed overflow map if the DMAP cannot completely serve. Unfortunately, two show-stopper bugs were discovered with this approach.
First, EDK-based firmware apparently requires that the runtime mapping exists simultaneously with the physical mapping for the SetVirtualAddressMap() call. Second, there were references from other open-source projects mentioning that some firmware required the presence of the physical mapping during the runtime call. Effectively, this forces both kernel and loader to provide both mappings for all runtime calls.
With such restrictions, informing the firmware about the details of the kernel address space only adds useless work. We could just as easily establish the 1:1 physical mapping during runtime and get rid of SetVirtualAddressMap() entirely. This approach was coded and the kernel interface to access runtime services is based on it.
During development, particularly when trying to make the loader modifications, it was quickly realized that there were no fault-reporting facilities in loader.efi. Machine exceptions resulted in a silent hang. Curiously, in such a situation the Intel firmware outputs the error code over the serial port over 115200/8/1 settings, regardless of UEFI console configuration, which was discovered by accident. Unfortunately, the error code alone is not enough to diagnose most problems.
A primitive fault reporter was written for loader.efi on amd64, which intercepts exceptions from the firmware IDT and dumps the machine state to the loader console. Due to the complexity of the interception and possible bugs which might do more harm than good there, the dumper is only activated on explicit administrator action.
Note that the described work only provides the kernel interfaces to make calling the EFI runtime services as easy as calling a regular C function. User-visible feature development making use of the new interfaces is being performed right now.
This project was sponsored by The FreeBSD Foundation.
Ports
KDE on FreeBSD
Contact: KDE on FreeBSD Team <[email protected]>
The KDE on FreeBSD team focuses on packaging the KDE software and making sure that the experience of KDE and Qt on FreeBSD is as good as possible.
The following big updates were landed in the ports tree this quarter:
- Added a Qt5 option to multimedia/mlt.
- Added the devel/grantlee5 port and, with it, Uses/grantlee.mk.
- Added the multimedia/gstreamer1-qt5 port.
- Added the net-im/telepathy-qt5 port.
- CMake was updated to versions 3.6.1 and 3.6.2.
- An important fix was made to qmake, where the clang version was not correctly detected.
- Qt 5.6.1 was committed to ports.
- Phonon and its backend to were updated to 4.9.0 in preparation for Qt 5.6.1.
- Updated the net-im/telepathy-qt4 port to 0.9.7.
- Various LibreSSL related fixes by Matthew Rezny.
- bsd.kde4.mk has been replaced by Uses/kde.mk.
- www/webkit-qt5 was fixed to depend on the systems leveldb.
In our development repository, we have done this work:
- The plasma5 branch has been kept up to date with KDE's upstream and contains ports for Frameworks 5.26.0, Plasma Desktop 5.8.0, and Applications 16.08.1 (branches/plasma5).
LXQt on FreeBSD
| Links | |
| FreeBSD LXQt Project | URL: https://wiki.FreeBSD.org/LXQt |
| LXQt Project | URL: http://lxqt.org/ |
| LXQt Development Repository | URL: https://www.assembla.com/spaces/lxqt/subversion/source |
Contact:
Olivier
Duchateau
<[email protected]>
Contact:
Jesper
Schmitz Mouridsen
<[email protected]>
LXQt is the Qt port of and the upcoming version of LXDE, the Lightweight Desktop Environment. It is the product of a merge between the LXDE-Qt and Razor-qt projects.
The porting effort remains very much a work in progress: it requires some components of Plasma 5, the new major KDE workspace.
The porting of the 0.11 branch is now complete, with new ports (compared to the previous release). See our wiki page for a complete list of applications.
We also have updates for:
- x11-toolkits/qtermwidget (0.7.0)
- x11/qterminal (0.7.0)
- x11/qterminal-l10n
Open tasks:
- Improve FreeBSD support in sysutils/lxqt-admin, especially with respect to user management.
- Add additional panel plugins.
Xfce on FreeBSD
| Links | |
| FreeBSD Xfce Project | URL: https://wiki.FreeBSD.org/Xfce |
| FreeBSD Xfce Repository | URL: https://www.assembla.com/spaces/xfce4/subversion/source |
Contact: FreeBSD Xfce Team <[email protected]>
Xfce is a free software desktop environment for Unix and Unix-like platforms such as FreeBSD. It aims to be fast and lightweight, while still being visually appealing and easy to use.
During this quarter, the team has kept these applications up-to-date:
- audio/xfmpc (0.2.3)
- deskutils/xfce4-notifyd (0.3.2)
- deskutils/xfce4-volumed-pulse (0.2.2)
- devel/thunar-vcs-plugin (0.1.5)
- misc/xfce4-weather-plugin (0.8.8)
- sysutils/xfce4-settings (4.12.1)
- x11/xfce4-clipman-plugin (1.4.0)
- x11/xfce4-dashboard (0.6.0)
- x11/xfce4-goodies, the meta-port for the Xfce4 Goodies Project (plugins, applications)
- x11/xfce4-whiskermenu-plugin (1.6.0)
We also follow the unstable releases; the current unstable release brings support for Gtk3 (available in our experimental repository) to:
- audio/xfce4-mpc-plugin (0.4.99)
- sysutils/garcon (0.5.0)
- sysutils/xfce4-battery-plugin (1.0.99)
- sysutils/xfce4-diskperf-plugin (2.5.99)
- sysutils/xfce4-fsguard-plugin (1.0.99)
- sysutils/xfce4-netload-plugin (1.2.99)
- sysutils/xfce4-systemload-plugin (1.1.99)
- www/xfce4-smartbookmark-plugin (0.4.99)
- x11/libexo (0.11.1)
- x11/libxfce4menu (4.13.1)
- x11/xfce4-dashboard (0.7.0)
- x11/xfce4-terminal (0.6.92)
- x11/xfce4-whiskermenu-plugin (2.0.1)
- x11-clocks/xfce4-datetime-plugin (0.6.99)
Currently the unstable releases work fine with our Gtk3 ports available in the ports tree, but in the future support for 3.18 will be removed in preference of 3.20.x.
Open tasks:
- Continue working on unstable releases.
Documentation
Documenting the History of Utilities in /bin and /sbin
Contact: Sevan Janiyan <[email protected]>
For EuroBSDcon, I began looking into inconsistencies within components inside our family of operating systems. My workflow consisted of reading the documentation for a given utility and checking the history in the revision control system for missing fixes or functionality in the trees of NetBSD, FreeBSD, OpenBSD, and DragonFly BSD.
One thing which became obvious very quickly was the inconsistency between operating systems about where and/or which version a utility originated in, despite our common heritage.
I began working through the man pages in FreeBSD, verifying the details in pages which already had a history section and making patches for those which did not.
From there, changes were propogated out to NetBSD, OpenBSD, and Dragonfly BSD where applicable (not all utilities originated from the same source or implementation, for example).
This was a good exercise in:
- Becoming familiar with mandoc.
- Using tools such as the linting functionality in mandoc and the igor documentation script.
- Becoming familiar with the locations where things are documented and with external sources of historical information, such as the BSD Family Tree included in the FreeBSD base system, and projects like The UNIX Heritage Society and the manual library on cat-v.org which hosts copies of manuals such as those shipped with Research UNIX. These manuals are not commonly available elsewhere.
Open tasks:
- Cover the remaining manuals for userland utilities, and maybe expand into library and syscall APIs, though I say that without estimating the feasibility. The history of components originating from a closed-source operating system is tricky to document, since older versions are not always available.
News Home | Status Home