Rendered at 21:08:23 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
zrobotics 22 hours ago [-]
This is extremely cool, I've played with FPGA design but never really even looked into custom ASIC design due to budget concerns. I hadn't heard of the tinytapeout project. Any recommended links for getting started?
I have a (really dumb, kinda meme-y) project involving a dead hakko soldering iron and "Internet of Soldering", a custom ASIC for the actual front-end would be the appropriate level of silly and possibly could fit in the project budget.
random__duck 21 hours ago [-]
As the designer of "an ethernet connected beacon that counts and never overflows until the heat death of the universe" (https://talesonthewire.com/projects/until_heat_death_do_us_p...) I believe you have found the perfect audience for a memeish hardware project. So don't expect anything but my upmost support. :)
> So for everyone else also running multiple instances of experimental hardware on their local network I have added an additional 2 Byte magic number using the all so original 0xCAFE (little endian) allowing packet parsers to distinguish between this ASIC’s experimental traffic and your other experimental traffic.
Just saying, it's good form to do network stuff in big endian. Although since it's already done I dunno how hard you should work to retcon that to 0xFECA...
inigyou 20 hours ago [-]
If your 0xCAFE gives you a 0xFECA1A55, stop going there...
toast0 18 hours ago [-]
Last time I eat vegetables..
zrobotics 19 hours ago [-]
Thanks for the pointers, hopefully I get some time to get started in that project soon!
Although I should point out that you missed the opportunity tor making the world's slowest counter and avoiding needing any expensive memory. Just declock it to an extreme degree. Hard to debug a design that never outputs anything for literal eons though.
Dylan16807 16 hours ago [-]
You could easily drop the clock to 32kHz but that only saves 10 bits out of nearly 400. Getting to 1Hz requires increasing amounts of analog circuitry and only saves 26 bits. Going many powers of 2 slower starts to require counters inside your clock, so now you're replacing expensive integrated bits with much more expensive discrete logic bits.
So, since clocking low can't save you many bits, might as well give it enough bits to handle any clock up to the limits of your transistors.
inigyou 20 hours ago [-]
Tiny Tapeout has its own guide for getting started
imtringued 13 hours ago [-]
The tiny tapeout project doesn't really fix the economics either. There's the obvious IO bottleneck, but worse than that, a 1000 LUT FPGA fits bigger designs and costs almost nothing. Like $2.50/FPGA at 1000 units.
roadbuster 23 hours ago [-]
Great project & article. Beyond the use of verilog, I couldn't find a description of the EDA stack. What tooling is used for the physical stages of design: place & route, LVS, DRC, and electrical circuit simulation (SPICE)?
This used the librelane classic flow, so it's a lot of verilator, yosys, openroad, opensta, magic and klayout. :)
inigyou 23 hours ago [-]
Nice! I've been following waferspace and tiny tapeout loosely but I'm sadly not in a position to be able to work on something right now.
I'm surprised you managed to fit something like this in a TT grid square.
random__duck 23 hours ago [-]
You can join multiple Tiny Tapeout tiles (grid squares) together to form bigger projects, which is what I did here. It's actually one of the bigger 4 grid projects.
Also how much you can fit really depends on the node the shuttle is targeting.
So I was in luck that this was on Global Foundries 180nm as you can cram in a lot for flip-flops onto a single tile compared with Skywater 130nm.
On the other hand the gates are much slower than on Skywater 130nm.
Hopefully both programs will be around for a while, so if you are still interested when you have more time you can join the shuttle then.
hasheddan 23 hours ago [-]
You can check out the tiles for this project and their exact location on the TT chip[0].
Nice, thanks for posting the link.
For the people unfamiliar with the Tiny Tapeout program: everyone that gets a copy of the Tiny Tapeout chip will be getting a copy of all the ASICs on the shuttle.
jandrese 21 hours ago [-]
That memory constraint is a real killer on the chip, even one that is a first generation toy like this. Only having 4 entries in the forwarding table is going to be an immediate problem if that switch is ever connected to another switch.
I wonder if it wouldn't make sense for one of the ports to be designated the "uplink" port and if the switch receives a packet destined for something not in the table it forwards it out the uplink port so long as the packet did not arrive via that port. It could even save a bit of memory by not storing entries for the uplink port in the table.
inigyou 20 hours ago [-]
Correct ethernet switch behavior forwards a packet to all ports if the destination MAC is unknown
jandrese 3 hours ago [-]
Which would work but in this case is not ideal because the majority of traffic arriving at the switch may not be on the table. Also, it will be constantly blowing away the existing entries in the table causing it to forget about the locally connected devices. Your switch would effectively turn into a hub. The uplink port solution is slightly more clumsy but hopefully prevents it from decaying to the worst case scenario most of the time.
crote 10 hours ago [-]
Sure, but at that point you're just building a complicated hub, and trying to actually operate it at speed would result in most packages getting dropped due to collisions as it is pretending to be full-duplex while actually only being half-duplex.
zamadatix 7 hours ago [-]
What do you mean by "pretending to be full duplex"?
inigyou 6 hours ago [-]
Modern Ethernet networks can be full-duplex because of all the buffering and separation between ports. If you're running ethernet on an actual shared medium like 10BASE5 or hubbed ethernet, it has to be half duplex to handle collisions better.
And the proposed switch design - transmitting most packets to most ports without buffering - is basically a hub. If you transmit two packets at the same time you will most likely create a collision. A real switch can buffer one of the packets.
inigyou 8 hours ago [-]
Ethernet switches are complicated hubs.
throw0101a 7 hours ago [-]
Some of the 'old' names:
> A network switch (also called switching hub, bridging hub, Ethernet switch, and—by the IEEE—MAC bridge[1]) is networking hardware that connects devices on a computer network by using packet switching to receive and forward data to the destination device.
On TinyTapeout you can add more tiles up to 16 times the size of this (at which point you likely jump to waferspace)
phire 16 hours ago [-]
There isn't really a restriction that a downstream device will only have one mac address.
It might be another switch. Even if it's not; VM/container networking is often implemented by bridging the host's NIC to a virtual switch, exposing the MAC addresses of all VMs/containers to the network.
jandrese 7 hours ago [-]
Right, which is why the tiny 4 entry MAC table might be a problem.
effnorwood 5 hours ago [-]
Make it into a company. Name it Sanfran.
noncoml 24 hours ago [-]
“ Cut-through vs Store-and-Forward”
I had a Google interview collapse because the last interviewer was a Principle Engineer who just couldn’t accept that a switch will ever forward a frame before reading in full and checking the FCS.
I was too much of an idiot to go along with it and instead ended up having an argument with him.
inigyou 23 hours ago [-]
Cut-through is applicable in a minority of scenarios which are sometimes common. The majority of switches, especially the ones that support any advanced features at all, can't use it. Even when you can use it, you still can't use it all the time because of port conflicts.
As far as advanced features go, even adding and removing VLAN tags is a headache in a cut-through switch.
userbinator 15 hours ago [-]
As far as advanced features go, even adding and removing VLAN tags is a headache in a cut-through switch.
Not really; it just needs to inject the tag right after the source address, or not pass it through at all; and of course recompute throughout and replace the FCS at the end. Other more advanced packet editing can be done in a streaming fashion too, with the latency only limited by the length of any "forward references" needed.
crote 10 hours ago [-]
But "just recompute the FCS" isn't quite the right approach, is it?
With a traditional cut-through switch the entire packet is passed as-is, so you're not modifying a corrupt package. You don't drop it, but the corruption is trivially detectable on the other side. If you naively recompute the FCS on transmit you essentially mark a corrupted package as valid, so you have to compute the FCS on transmit and receive, and intentionally send out the wrong FCS on transmit if you determine the receive FCS is invalid.
bcrl 7 hours ago [-]
It's easy enough to write a bit of Verilog to perform partial updates on the CRC to only compensate for what has changed. I did so for an FPGA project I implemented a number of years ago when processing VLAN tags and doing IP forwarding.
That said, there are lazy hardware designers that just slap on a block to recompute the CRC and replace the old one with the new. An experience I had years ago at Red Hat taught me that the hard way as a shiny new Cisco ethernet switch IT deployed ended up costing us weeks of frustration and delayed a release when it caused bit flips in NFS packets during stress testing. The switch was so awesomely advanced that it recomputed the CRC on the bit flipped packets causing kernel builds to fail. <sigh>
foobiekr 5 hours ago [-]
Wow that rings a bell.
The lack of end to end ECC protection in some packet paths on very high end ASICs is something I just can’t get over.
random__duck 24 hours ago [-]
Sounds like they could have used you on that team.
Bad luck, but it at least you can walk away proud knowing it was not an issue with your technical skills. :)
Aurornis 16 hours ago [-]
I remember studying cut-through forwarding in a class and thinking it was obviously going to be used everywhere.
Then I was disappointed to enter the real world of networking and learn that it wasn’t actually used as often as I had been led to believe due to conflicting with higher priority features.
So the interview may have been a conflict between textbook learning and the Google engineer’s practical experience. It’s tough to encounter that conflict in interviews because you get two people talking past each other.
sophacles 22 hours ago [-]
I'm struggling to think of any real-world networking scenario where cut-through switching for ethernet can provide any sort of benefit.
Most data-center networks are involve aggregation/spine/core switches at higher speed than TOR/leaf switches - so the packet has to be buffered.
Most every workload is going to involve port collisions (2 packets at the same time needing the same output port) - again a full buffering required (or drops and throughput reduction).
Bigger networks start using vlans, mpls, ip routing at switches, and vxlan - requiring more lookups per-packet, larger "minimum chunks of packet seen before port is determined" sizes, etc reducing the value of cut-through.
And so on.
But i also struggle to find a scenario where a principle engineer at google is refusing to accept that cut-through switches exist, but rather, was refusing to accept that the scenario they're talking through with you would ever involve an effectively deployed cut-through switch (since that's basically never).
BornToLag 22 hours ago [-]
Ethernet frame preemption has been standardized for about 10 years for ultra low latency applications. See 802.1Qbu:
"Defines a class of service for time-critical frames that requests the transmitter in a bridged Local Area Network to suspend the transmission of a non-time-critical frame, and allow for one or more time-critical frames to be transmitted. When the time-critical frames have been transmitted, the transmission of the preempted frame is resumed. A non-time-critical frame could be preempted multiple times."
toast0 21 hours ago [-]
> I'm struggling to think of any real-world networking scenario where cut-through switching for ethernet can provide any sort of benefit.
When it works, it reduces latency and reduces buffer memory usage a smidge. This ASIC switch can't afford to buffer any packets, so cut through is the only option... AFAIK, most ethernet switches have at least some buffer memory...
> Most data-center networks are involve aggregation/spine/core switches at higher speed than TOR/leaf switches - so the packet has to be buffered.
If you have traffic flowing within a rack, you can cut through for that where the machines attach to the TOR switch at the same speed.
If you have traffic between racks, and it goes through a spine, it likely needs to buffer at the TORs, but it could cut through at the spine.
> Most every workload is going to involve port collisions (2 packets at the same time needing the same output port) - again a full buffering required (or drops and throughput reduction).
Depends on the workload and redundancy requirements. If you want to have dual link aggregation and no loss of throughput with a single lost link, you'll upgrade line speed when link utilization is at 50%, so maybe 50% of packets could be cut through (depending on burstiness). If latency is a key driver, you may have line speed much higher than throughput and have very little utilization.
tuetuopay 21 hours ago [-]
The two scenarios I can think of that definitely make use of cut-through are timekeeping systems, and HFT. The latter goes even further, where the switches at the exchange do cut-through, but also ensure the packet is broadcast to multiple ports at the exact same time. This is done so that no specific HFT firm get an advantage from being on port 10 rather than port 20. They even measure out the outgoing fibers to be the same length.
empty_set35204 18 hours ago [-]
The amount of effort an exchange goes through to ensure fairness varies substantially, unfortunately. Many do have measurable and consistent /semi-consistent differences between gateways that can be attributed to network asymmetries.
noncoml 22 hours ago [-]
> but rather, was refusing to accept that the scenario they're talking through with you would ever involve an effectively deployed cut-through switch (since that's basically never).
Nope, the argument was exactly if a cut-through switch exists or ever existed
Edit: Actually the conversation came back to me:
He asked me about the frame format. Once I drew that on the board, he asked why is the DST before the SRC. My answer was in order for the switch to start doing the lookups ASAP and that's when I mentioned that a cut-through switch can switch the packet before it even finishes receiving it.
After that we spend the rest of the interview time arguing if cut-through switches actually exist or not. Both he and I having too big of an ego to let it go..
icedchai 22 hours ago [-]
So he didn't want to google it?
inigyou 20 hours ago [-]
If you Google it, you find many copies of theory, but not a single reference to a cut-through switch ever existing in real life.
“Arista data planes support these packet forwarding modes:
Store and forward: the switch accumulates entire packets before forwarding them.
Cut through: the switch begins forwarding frames before their reception is complete.”
stinkbeetle 20 hours ago [-]
I see you've never had a run-in with a blowhard in a position of power. Lucky.
icedchai 19 hours ago [-]
I've had it happen in an interview, many years ago. I told him we'd have to agree to disagree. After I was hired, he admitted he looked it up and I was correct (some weird SQL question he asked.)
userbinator 15 hours ago [-]
After I was hired, he admitted he looked it up and I was correct
The best possible outcome.
noncoml 19 hours ago [-]
“So he didn't want to google it?”
No, he didn’t want interrupt the interview to start checking out phones.
“I told him we'd have to agree to disagree.”
That’s where I went wrong. But I like to think learned my lesson.
brcmthrowaway 22 hours ago [-]
What do HFT switches do?
gjvc 22 hours ago [-]
"principal"
netik 14 hours ago [-]
How much of this is AI?
random__duck 8 hours ago [-]
No AI was used either for the project itself or for the post.
I encourage you to look at the code and the git history.
brcmthrowaway 22 hours ago [-]
Was Claude used?
random__duck 22 hours ago [-]
No AI was used either for the project itself or for the post.
I have a (really dumb, kinda meme-y) project involving a dead hakko soldering iron and "Internet of Soldering", a custom ASIC for the actual front-end would be the appropriate level of silly and possibly could fit in the project budget.
I detailed the roadmap for custom ASIC design a bit here: https://talesonthewire.com/projects/two_weeks_until_tapeout/... and joining the Tiny Tapeout discord community is a great place to reach out for help.
Just saying, it's good form to do network stuff in big endian. Although since it's already done I dunno how hard you should work to retcon that to 0xFECA...
Although I should point out that you missed the opportunity tor making the world's slowest counter and avoiding needing any expensive memory. Just declock it to an extreme degree. Hard to debug a design that never outputs anything for literal eons though.
So, since clocking low can't save you many bits, might as well give it enough bits to handle any clock up to the limits of your transistors.
This used the librelane classic flow, so it's a lot of verilator, yosys, openroad, opensta, magic and klayout. :)
I'm surprised you managed to fit something like this in a TT grid square.
Also how much you can fit really depends on the node the shuttle is targeting. So I was in luck that this was on Global Foundries 180nm as you can cram in a lot for flip-flops onto a single tile compared with Skywater 130nm. On the other hand the gates are much slower than on Skywater 130nm.
Hopefully both programs will be around for a while, so if you are still interested when you have more time you can join the shuttle then.
[0] https://tinytapeout.com/chips/ttgf26b/tt_um_coffeepot
I wonder if it wouldn't make sense for one of the ports to be designated the "uplink" port and if the switch receives a packet destined for something not in the table it forwards it out the uplink port so long as the packet did not arrive via that port. It could even save a bit of memory by not storing entries for the uplink port in the table.
And the proposed switch design - transmitting most packets to most ports without buffering - is basically a hub. If you transmit two packets at the same time you will most likely create a collision. A real switch can buffer one of the packets.
> A network switch (also called switching hub, bridging hub, Ethernet switch, and—by the IEEE—MAC bridge[1]) is networking hardware that connects devices on a computer network by using packet switching to receive and forward data to the destination device.
* https://en.wikipedia.org/wiki/Network_switch
It might be another switch. Even if it's not; VM/container networking is often implemented by bridging the host's NIC to a virtual switch, exposing the MAC addresses of all VMs/containers to the network.
I had a Google interview collapse because the last interviewer was a Principle Engineer who just couldn’t accept that a switch will ever forward a frame before reading in full and checking the FCS.
I was too much of an idiot to go along with it and instead ended up having an argument with him.
As far as advanced features go, even adding and removing VLAN tags is a headache in a cut-through switch.
Not really; it just needs to inject the tag right after the source address, or not pass it through at all; and of course recompute throughout and replace the FCS at the end. Other more advanced packet editing can be done in a streaming fashion too, with the latency only limited by the length of any "forward references" needed.
With a traditional cut-through switch the entire packet is passed as-is, so you're not modifying a corrupt package. You don't drop it, but the corruption is trivially detectable on the other side. If you naively recompute the FCS on transmit you essentially mark a corrupted package as valid, so you have to compute the FCS on transmit and receive, and intentionally send out the wrong FCS on transmit if you determine the receive FCS is invalid.
That said, there are lazy hardware designers that just slap on a block to recompute the CRC and replace the old one with the new. An experience I had years ago at Red Hat taught me that the hard way as a shiny new Cisco ethernet switch IT deployed ended up costing us weeks of frustration and delayed a release when it caused bit flips in NFS packets during stress testing. The switch was so awesomely advanced that it recomputed the CRC on the bit flipped packets causing kernel builds to fail. <sigh>
The lack of end to end ECC protection in some packet paths on very high end ASICs is something I just can’t get over.
Then I was disappointed to enter the real world of networking and learn that it wasn’t actually used as often as I had been led to believe due to conflicting with higher priority features.
So the interview may have been a conflict between textbook learning and the Google engineer’s practical experience. It’s tough to encounter that conflict in interviews because you get two people talking past each other.
Most data-center networks are involve aggregation/spine/core switches at higher speed than TOR/leaf switches - so the packet has to be buffered.
Most every workload is going to involve port collisions (2 packets at the same time needing the same output port) - again a full buffering required (or drops and throughput reduction).
Bigger networks start using vlans, mpls, ip routing at switches, and vxlan - requiring more lookups per-packet, larger "minimum chunks of packet seen before port is determined" sizes, etc reducing the value of cut-through.
And so on.
But i also struggle to find a scenario where a principle engineer at google is refusing to accept that cut-through switches exist, but rather, was refusing to accept that the scenario they're talking through with you would ever involve an effectively deployed cut-through switch (since that's basically never).
"Defines a class of service for time-critical frames that requests the transmitter in a bridged Local Area Network to suspend the transmission of a non-time-critical frame, and allow for one or more time-critical frames to be transmitted. When the time-critical frames have been transmitted, the transmission of the preempted frame is resumed. A non-time-critical frame could be preempted multiple times."
When it works, it reduces latency and reduces buffer memory usage a smidge. This ASIC switch can't afford to buffer any packets, so cut through is the only option... AFAIK, most ethernet switches have at least some buffer memory...
> Most data-center networks are involve aggregation/spine/core switches at higher speed than TOR/leaf switches - so the packet has to be buffered.
If you have traffic flowing within a rack, you can cut through for that where the machines attach to the TOR switch at the same speed.
If you have traffic between racks, and it goes through a spine, it likely needs to buffer at the TORs, but it could cut through at the spine.
> Most every workload is going to involve port collisions (2 packets at the same time needing the same output port) - again a full buffering required (or drops and throughput reduction).
Depends on the workload and redundancy requirements. If you want to have dual link aggregation and no loss of throughput with a single lost link, you'll upgrade line speed when link utilization is at 50%, so maybe 50% of packets could be cut through (depending on burstiness). If latency is a key driver, you may have line speed much higher than throughput and have very little utilization.
Nope, the argument was exactly if a cut-through switch exists or ever existed
Edit: Actually the conversation came back to me:
He asked me about the frame format. Once I drew that on the board, he asked why is the DST before the SRC. My answer was in order for the switch to start doing the lookups ASAP and that's when I mentioned that a cut-through switch can switch the packet before it even finishes receiving it.
After that we spend the rest of the interview time arguing if cut-through switches actually exist or not. Both he and I having too big of an ego to let it go..
Cisco:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/ne...
HPE Aruba:
https://arubanetworking.hpe.com/techdocs/AOS-CX/AOSCX-CLI-Ba...
Fortinet:
https://docs.fortinet.com/document/fortiswitch/7.2.7/adminis...
Arista:
https://arista.my.site.com/AristaCommunity/s/question/0D55w0...
Nvidia/Mellanox:
https://enterprise-support.nvidia.com/s/article/switch-forwa...
Extreme Networks:
https://documentation.extremenetworks.com/Switch%20Engine%20...
“Arista data planes support these packet forwarding modes: Store and forward: the switch accumulates entire packets before forwarding them. Cut through: the switch begins forwarding frames before their reception is complete.”
The best possible outcome.
No, he didn’t want interrupt the interview to start checking out phones.
“I told him we'd have to agree to disagree.”
That’s where I went wrong. But I like to think learned my lesson.