Usenix 2002 FreeBSD Developer Summit III

rwatson : What about userland?

julian : It can run different threads in userland. The primitives are all there it just needs a bit more help. I would like to put an idea out. Is it a good idea to be able to have non-threaded programs linking with threaded libraries?

rwatson : Putting async I/O into such a thing would make sense.

julian : The library would not care who was accessing it.

rwatson : For instance libc could be threaded or not.

julian : That would be interesting. I don't know if the two interfaces are incompatible.

jhb : X does this.

dillon : It is very doable but you have to make it non-preemptive. If you're switching non-preemptively you can use library routines which are non threaded.

julian : If I do what I'm thinking of doing then each lib will have its own KSE group.

dillon : stdio does not have to be thread aware if you don't schedule preemptively. It all matters where it blocks.

julian : Since you're a non-threaded program you don't know that.

rwatson : If you're going to support that, libc has to support threads.

rwatson : It sounds like some complexity goes away. Can we use 1 libc with has threading?

julian : Do we want to go down this path?

rwatson : Now or later?

julian : What do I design now to do this?

jhb : For example libc_r does not work with rfork.

julian : The answer is that yes we should move forward. Tricky issues, signals...

warner : Have people talked about pthread programs and cancellation points?

julian : The pthreads library does not assume that you're only going to change threads at yield() points. We are going to have cancellation points. There is an unimplemented call which will be able to send a thread targeted signal even into the kernel.

julian : When a thread is scheduled onto a KSE there is a mailbox that the userland thread scheduler updates.

julian : Is there anyone else who has some time or test it? How many people should test this before I check it in? There is a patch that's continuously updated on my web site to be able to patch it to -CURRENT. There is a CVSUP target from cvsup 10 which will bring down the sources. If you go to my web page on freefal there is a pointer there to a web page that explains how to CVSUP from source.

rwatson : What about SMP locking for this?

julian : Handled by the proc locking. Has not been tried on SMP machines yet.

obrien : What about on Sparc?

julian : You may need to stub things out.

jhb : Is the paper on the web site?

julian : The updated copy has disappeared.

unknown : What's the different between NetBSD and FreeBSD on this?

julian : Logically not a tremendous difference but Net follows the paper closely and Free takes the idea and makes it into a production system. There were some tough battles on -arch about this. The tricky point is that the proc structure has to be broken into 4 instead of 2. If you want to be able to do POSIX you need to be able to treated as different processes but in other systems you need to group the threads. You wind up with two classes of threads. You need another structure to do the aggregation. In the end we ended up breaking up the proc structure into 4 pieces to not overwhelm the CPU when scheduling threads. This is the major difference.

julian : I greatly admire the NetBSD way which is to take an idea and not dilute it.

julian : Net is also putting a Solaris compatible threads package on top of their scheduler activations in the Solaris ABI.

jhb : Yesterday we talked about SMP related things so I'll give a summary and then give a list of things for 5.0.

jhb : The big thing for 5.0 is to get the network stack out from under Giant.

jhb : Jeffrey Hsu and Jennifer Yang were here to talk about this. They have the PCBs checked in now.

jennifer : Interface Queues and SynCache might be done.

jhb : Aside from network the newbus locking needs to be done (Warner Losh) and also CAM stuff. No known status on CAM. Perhaps CAM is not needed for 5.0

jhb : Disk drive interrupts? Would help performance. Going to talk to Poul Henning-Kamp

jhb : Alan Cox is working on the VM system. Working based on the old Mach stuff. Objective for 5.0 is to get zero fill and execute on write to work without Giant. In future he wants to look at locking down pmap() functions.

jhb : Still some stability issues. UMA breaks some assumptions. For instance sockets assume that once memory is a socket its a socket forever, this is no longer true.

jhb : Talked to Mike Smith about 5.0 and have decided to stop adding features so that we can start clean up 5.0 and make it a real release. This might require hacks.

rwatson : For example in the UMA case there could be a flag to just say "don't reclaim this zone" -- this would help with issues such as the socket code assuming memory is type stable.

Over to Alan Cox on the VM system.

alc : Nothing to say.

bmilekic : As much as I might get hated for this. Will preemption stuff go in by 5.0?

jhb :No, that's a 6.0 thing. There are things to do first.

unknown : Could this come in in the life time of 5.? 5.1?

rwatson : This is a release issue really.

jhb : Yes, the kernel is pre-emptive.

rwatson : Perhaps we should talk about is performance goals? What are the comparisons to make? Perhaps head of 4 with head of 5. We'll see a mix.

jhb : I need to run benchmarks.

rwatson : In terms of SMP features when will VM be ready to be measured? I can't put a date on it.

alc : I think I told John was in time for release. I'm already doing performance testing so we've already started.

rwatson : We'll pick a date to start doing measurements. Perhaps 2 or 3 weeks from now.

alc : My guess is the the locking pmap is going to take some time to shake out. On the other hand the next major module we should be working on is machine dependent level. Last we should try approaching the vmobject level. I'll start worrying about performance in the near term.

rwatson : Will threading improve latency or throughput for networking?

bmilekic : I would like if we could actually start before.

rwatson : Do you have a timeline for the interrupt threading stuff?

bmilekic : I finished some things last night but there are still issues. In a couple of weeks it should be ready for first commit.

rwatson : Informally beginning to measure performance now. What are the right sets of tests? Need to discuss on -arch.

alc : It would be nice to have that committed to the tools directory.

jhb : The statistics analysis package are we using.

bmilekic : I have some good success with netpipe for overall measurement.

rwatson : Need to be using consistent compilers because of the compiler change. Also all our debugging stuff will slow down the benchmarking.

dillon : Debug stuff on 5.0. I think it might be reasonable then to take the space hit and always have the debugging in but turn it on and off with sysctl.

rwatson : We should commit an optimized kernel configuration and benchmarking guidelines to the tree as well.

rwatson : I think we should continue the performance discussion. We want to accomplish a couple of things. One is stability measurement. What are the things we need to be measuring? What is our definition of useful?

hsu : End to end measurement with gigabit cards. For latency test connections per second. Can use ttcp or netbench in ports.

gnn : need to make sure we run against all of 4.6

julian : Need to really have 3 tests. 4.6 (forever) 4.x (following updates) and -CURRENT.

rwatson : There are other dimensions. Degree of parallelism for instance. We might see degradation in uni but get good stuff in multi case.

julian : Test for impact of KSE complications as well.

alfred : I think as the results come through people should not be too worried about it. Perhaps we should benchmark database type stuff as well. Need to do something comprehensive.

obrien : What does the test matrix look like? Different architectures and different numbers of processors.

rwatson : Can we make a multi-processor run uni-processor.

alfred : Run queue and scheduler stuff?

julian : Will talk to Alfred.

rwatson : Is scalability testing important?

obrienM : NFS testing.

rwatson : What about UI testing?

hsu : x11perf is the way to do that.

dillon : Currently we have a directory for regression tests, should we do one for performance tests?

gnn : talk to sleepycat for DB tests, see if they have some

alfred : Really nice to tests DB applications that are heavily thread dependent.

hsu :Apache 2 has threads.

rwatson : What about commercial folks? What do you do.

ps : Normally what we end up doing is using the snapshot on some machines and see if the bugs are out. There is no performance testing really.

rwatson : Again, what about performance?

ps : We've really never had one. It's more just bugs. We've just never found the performance to be a problem.

rwatson : We need to create a forum for talking about performance. We need reproducible test cases.

ps : There's also other things. We've been doing lots of looking at this. FreeBSD gets kicked down by attacks for instance. We have a lot of tools to get to the project though.

rwatson : I will set up the mailing list.

jhb : Questions about alpha?

rwatson : KSE on alpha?

julian : We have patches so it compiles and runs non-KSE programs. You can have the patched version of the alpha kernel up and running though.

rwatson : Is the task owned of making this work on Alpha?

dfr : It works as far as I get to use it. It's not used in production right now.

ps : Intel shipped me a quad processor IA64 board. (McKinley is the name of the board).

rwatson : What does it need for 5.0?

dfr : It works, it works for SMP. Self hosts, build worlds. sysinstall compiles but needs more kicking to work.

ps : Intel wants us to ship a CD.

dfr : There is no thread support right now (threading library needs to move to get/setcontext rather than longjmp).

dfr : Need to move every driver to use BUS DMA for large memory machines to get bounce buffers.

jhb : PHK is working on using a new libwhisk so that sysinstall et al work on all systems.

jake : Take control of KSE stuff on Sparc 64.

rwatson : Do we have a Sparc 64 in the cluster?

jake : It's not in the cluster yet. It's a serial cluster issue.

rwatson : Package building on S64?

jake : Perhaps a bunch of Ultra 60s for a package build.

obrien : 1500 build right now?

jake : Yes, but a lot of the same bug in packages are broken.

jhb : Timeline for X?

jake : Not really.

rwatson : In terms of 5.0 how comfortable are you?

jake : sysinstall is the only problem.

benno : I got it up to execing a fake init in the simulator and printing "hello world". Trying to work with real hardware. I now have some semblance of busdma and am working on other stuff. GEM on iMac is in an embryonic state. Should get to NFS mount in a few weeks.

rwatson : How do you feel about your timeline?

benno : I'm not sure we'll have something fully workable for 5.0.

rwatson : You're not at the point yet on working on KSE are you?

benno : No, need a useful system first.

rwatson : I know that we're having simulator problems.

obrien : The issues are about legal and NDA. AMD decided on FreeBSD Mall as the NDA person. I have not had a working simulator since September.

ps : I can make that happen, as well as real hardware.

obrien :I've got a cross tool chain in the tree. I have some untested pmap stuff. Hardware has been available for a month or so. We could boot FreeBSD 4.6 today if only we had hardware.

rwatson : What do you think about 5.0? Should we discuss that at another date?

unknown :Juniper offered.

obrien : But we have no hardware.

unknown :Juniper thinks it's OK but doesn't want to have it rot in the tree.

brooks : I have a line on a company that does compact PCI with R6Ks.

rwatson : We're waiting for someone to turn up.

julian : Are the labels the same on all structures?

rwatson : You can modify this but there are issues with memory: is the space needed for a label too large to add to an mbuf header, for example? The label is small, but there area lot of them?

bmilekic : When you're freeing the mbuf do you write the label data?

rwatson : We blank it when we free it.

bmilekic : I do not think the 36 bytes in the mbuf header is a problem.

julian : I'm more interested in the "why" than the how.

rwatson : A lot of people are interested in this. Some of the things that do interest a lot of people are things like doing on the fly security for a web server.

julian : Is there a black hatted TLA interested?

rwatson : Yes and several gov'ts. As well as plenty of financial folks.

rwatson : There's a lot of userland stuff that's not done yet.

murray : Release engineering area of the web site www.FreeBSD.org/releng. For DP2 question about p4 or CVS? Will probably use p4 for DP2 as well. USB subsystem? Perl removal? KSE?

julian : KSE should be able to run simple tests.

obrien : Is whatever you have committed by DP2 be the same as the release.

julian : It will be a subset.

murray : What will the status be of KSE in userland for 5.0?

julian : Can't answer that right now. We're not removing the old libraries. The userland work will happen between DP2 and release. The next step is MP as well as UP.

obrien : Are we heading for a release?

murray : yes.

obrien : Then we have to stop having major commits.

murray : Yes, the discussion today is what are the major must have features.

rwatson : We need to decide if there are major upcoming problems and reduce risk on things like KSE.

julian : That's why I want to get milestone 3 in now.

rwatson : Do you think that KSE related changes from later milestones are going to be isolated to KSE or pervasive?

julian : Hard to say. My guess is that milestone 4 stuff should be less pervasive.

rwatson : What happens if KSE just doesn't work?

julian : Well it does work, the patches work, it's a question of risk. We need to check on new things, like locking two threads in the same process.

dillon : KSEs only become fragile when pthread uses them. That's the turning point.

obrien : I'd like the rules for the rest of the summer, I hope we'll talk about that.

murray : Earlier is better.

mini : I think the cutoff point for KSE might be milestone 3.

rwatson : It's the kind of thing where if we need to back out we can.

julian : If you're not going to run KSEs then you're OK.

rwatson : I think it's low risk. Let's avoid the risk is the message.

julian : The next DP2 (where we'd like milestone 4).

alfred : We really need KSE so all this concern about stuff that no one really uses is not a big deal. People just need to play catch up. We have performance problems and we need to solve those.

obrien : We quickly need to figure out our policy on multiple archs.

rwatson : I briefly want to respond to Alfred. We have asserted that KSE will be experimental. It will be in and 5.0 will go out but there might be issues.

jhb : Realistically for the network stack is that IPv4 sockets will not be giant. But this is only in the network stack world. Several people are working on it.

rwatson : The GEOM stuff will be enabled by default in 5.0. Sparc depends on it. I do not know what the impediments are to that though.

julian : The kernel stuff is there but the user space is not. It can't become the default until everything is there.

warner : What level of control are you going to exercise over the tree in the coming months?

murray : You're going to see more level of control but we expect the requests to be reasonable. It's a very open process.

jhb : How are we going to address the 5/6 split?

murray : Carefully is all I can say.

rwatson : For 5. 0 we need to have a more informed decision. The release engineers will be trying to reduce the number of large code changes more as time goes by. We don't have to wait for 5.x to be perfectly stable before we branch.

murray : Let's move it to more general discussion of DP2? Specific technologies.

bmilekic : Is there a strategy to lock other protocols that are not locked down onw?

obrien : How much more do we need to do before 5.0?

jhb : Bug fixing is what we're doing.

rwatson : The answer on the network stack. We need to choose a strategy on how to handle the other protocols.

obrien : The crux is that socket locking must be in 5.0.

rwatson : There are 2 or 3 problems. Routing code is a problem. See earlier discussions.

dfr : RCng is essentially done. What it needs is testers.

alfred : What about libh (I think libh is wrong but this is what I heard)?

jhb : It's very far along but not a 5.0 thing.

warner : Problems with interrupt routing in ACPI?

takawata : Cannot handle PCI<->PCI interrupt routing. Many 802.11x have this problem.

julian : Is it a problem from Intel?

takawata : This is not an Intel problem but a problem on our side. PCI<->PCI routing code should be added. New code is necessary.

Whiteboard

UFS2		rcNG		KSE M3			CAM SMPng

GEOM		TrustedBSD Malc	BusDMA			Newbus SMPng

C++		Cardbus		libwhisk/sysinstall	KOBJ? (no!)
				sparc64

Perl Removal	ACPI		Alpha SMP Stability	Pkgs for
							sparc64, IA64

devd		PCI intr route	document hints		release docs
							for new
							platform

unknown : Firewire?

rwatson : What hardware shipping on IA64?

dfr : Intel stuff

rwatson : What about on Sparc64?

obrien : Very limited (hme...)

rwatson : KOBJ extensions discussed at BSDCon?

warner : Not sure, probably not for 5.0. Pervasive, so no.

rwatson : How broken is C++?

obrien : Only on sparc64. Don't really know yet, but it's probably a library issue. The compiler is a pre-release snapshot. The diffs are now getting large from May 5 to now. We should attempt to be as far along this gcc branch as possible come release.