Thought you guys might appreciate some geekery.

A General Discussion forum for cars and other topics, and a great place to introduce yourself if you are new to NICO!
User avatar
MinisterofDOOM
Moderator
Posts: 30928
Joined: Wed May 19, 2004 5:51 pm
Car: 1962 Corvair Monza
1961 Corvair Lakewood
1974 Unimog 404
1997 Pathfinder XE
2005 Lincoln LS8
Former:
1995 Q45t
1993 Maxima GXE
1995 Ranger XL 2.3
1984 Coupe DeVille
Location: The middle of nowhere.

Post

I have recently put some time, money, and luck in obtaining cheap, slightly-obsolete equipment into building a home network that's a little more robust than just Centurylink's abysmal Zyxel modem/router/AP combo unit.

I am a bit of a geek, and have about sixty-seven trillion networked devices in my house. A lot of them are handhelds that lack wired networking support, but many others have at least 100M ethernet ports.

I have a split-level home, and there's a half-height crawlspace under the main level. It's cool, unfinished, and mostly empty. I decided I'd use that for my networking center. I ran CAT6 stretches from that room out into the house. So far, I've got 9 runs (including the RJ11-capped DSL source for the modem) and will probably add 4-6 more to round things out on the second floor later on.

In my office, I have 3 PCs and a laserjet printer.

In the TV room, I have several consoles, a Nexus Player, a Steam Link, a "smart" BluRay player, and the dock for my Surface Pro 4. The majority of the consoles are last-gen or much earlier, so no wired networking is needed for them. The Nexus Player supports ethernet via USB adapter (daisy-chained through an OTG adapter) and negates the need for networking to the BluRay player. And the Wii U only works with Nintendo's branded, overpriced, impossible-to-find USB-to-Ethernet adapter, so it's on wifi despite my desire to keep it wired.

In the garage, I have an old Dell workstation that's currently fed by wifi serving as a shop computer. I have an utterly ABYSMAL cheapo euro-layout keyboard that came with a PC case years ago, with a sticky right shift key and stupid LED lighting that I don't care about getting all greasy. Nice for looking up pictures when working on unfamiliar cars. That machine will move to ethernet soon, but I'm not in the mood for running that cable at the moment, so it'll wait.

Upstairs, there's a Roku 3 and a pair of Raspberry Pis serving media duties, 3 DSs of varying generations, and a couple laptops. All of this is on Wifi right now, but I'll run ethernet up there in the future. Rokus work much better over wired connections in my experience.

Then, in the crawlspace, I have an old Linksys SGE2000 24-port stackable router, the aforementioned antique Centurylink Router, an old non-AC Netgear wireless AP (I know, I need to get something AC cabable now that the wired side is Gigabit) and two old Dell servers (an R180 with a single hyperthreaded quad-core Xeon, and a 2950 with non-HT dual cores, both with 16 gigs of RAM, and with about 15 TB of SAS drives between them before RAID overhead, and booting ESXi 6.0). Oh, and a really crappy old 13 inch monitor and a random USB keyboard serving KVM duties. I'm trying to get my hands on a cheap old rack KVM to cut down on same cable clutter, but it's not a particularly urgent need.

The switch is set up like this:
Image

A few of the devices on the network are limited by 10/100 NICs (the Nexus Player disappointingly among them), but everything else is working wonderfully at 1 gig. My switch actually has 4 SFP ports if I wanted to get really crazy, but it's only listed as a 10/100/1000 switch, so I'm not sure if it could even utilize any of SFPs extra capacity.

The really depressing part of this is that all of this connects outside through a nominally 50Mb ADSL connection. Nominally, but really more like 30Mb. It's all I can get out here, and I cannot possibly say enough horrible, awful, unkind things about Centurylink to express my true feelings. However, part of the purpose of this arrangement is to circumvent that slow internet connection by housing regularly-used things like ISOs, installers, and media files on a local NAS, so I can grab them rapidly on the network and make new-machine deployment very quick and efficient.

One of the most enjoyable things about running a pair of ESXi servers is that I can play with new and different operating systems without having to fart around with dual-booting on existing workhorse machines. No Grub, no UEFI/Secureboot nonsense. Just deploy a new VM, install the OS, and see what I think. I'm growing fond of BSD and CentOS, especially in lightweight formats.

Here's my office setup:
Image
(Yes, like I said: geek. Also, that Lego supercar in the background? Working 4-speed manual transmission. Cooler than your daily driver.)

And the...er..."rack."
Image


ArmedAviator
Posts: 526
Joined: Tue Mar 22, 2016 5:28 pm
Car: 2012 Infiniti M37x
Location: SW Ohio

Post

I have had been experimenting with networking stuff over the past few years as well.

Currently, I am running the following geek setup....
50Mbit/5Mbit Time Warner Cable connected through a self-provided Motorola Surfboard DOCSIS 3.0 modem.

The modem is connected to a 10/100 port on the motherboard of a repurposed 1U Lacie EthernetDisk (VIA 1000MHz processor) running the latest version of pfSense. I put an external 10/100/1000 card into the pfSense box which is connected to a Netgear GS724TR gigabit switch. The switch and pfSense have configured VLANS.

VLAN0 (default) = 10.1.1.0/24 (for administration only)
VLAN10 = 10.1.10.0/24 (shared to all other VLANs such as NAS, Smart TVs, and gaming consoles)
VLAN20 = 10.1.20.0/24 (my stuff)
VLAN30 = 10.1.30.0/24 (roommate 1)
VLAN40 = 10.1.40.0/24 (roommate 2)
VLAN50 = 10.1.50.0/24 etc....
VLAN60 = 10.1.60.0/24

The pfSense box is the house's gateway, firewall, DHCP, NTP, and DNS server, and traffic shaper. The traffic shaper is configured to allow any of the VLANs maximum throughput of the 50/5 cable connection, but every VLAN is guaranteed atleast 1/5th of the link bandwidth (so if every VLAN was downloading full bore, every VLAN would get a fair share).

Each person in the house (5 roomates) gets their own VLAN - hence the traffic shaping. Each VLAN also has it's own WiFi AP (as we all had one each before we moved in together) so none of them are overloaded with wireless devices, each is on their respective VLAN, and I configured them so none of the 5GHz or 2.4GHz channels are interfering with eachother.

Unfortunately this house is not wired with ethernet so I used MoCa 2.0 devices and some careful wiring hiding to make this work with wired devices.

I used 2 seperate, non-interfering MoCa 2.0 networks running through the house. One goes from the server room to the top floor and is connected to one roommate's WiFi/switch and desktop and is configured to his VLAN. The other runs from the server room to the main floor living room to a "dumb" switch connected to a few gaming consoles and the Smart TV on VLAN10.

To get the network to my room and the other 4 WiFi APs on the main floor (still on their respective VLANs), I was able to snake up 2 optical cables from the server room to a mostly-unused closet between the kitchen and my room. It's only about 30' of cable (times 2) and went straight up through the return vents. They are connected with an identical Netgear switch as before and with the same tagged VLANs configured so packets are routed accordingly (on the same subnet). I have the 2 optical cables in a round-robin type LAG so they communicate as a single connection but with the throughput of 2 gigabit connections. This also allows for redundancy incase one becomes disconnected or fails.

Connected to this switch is 2 ethernet cables to my desktop, also in a round-robin LAG and each of the 4 additional WiFi APs on their respective VLANs.

My Gentoo Linux NAS is home built with 6x 4TB drives in RAID10 (was in RAID5 until last week's big bug-find in the RAID56 code in Btrfs) containing a ton of movies, music, and other personal data. All is shared as-needed through special permissions in Samba and rsync. Currently my desktop has a second copy of all of the NAS data as a backup, but this should be getting a second dedicated 4U rackmount case and newer internals to quiet down my room and reduce heat. The NAS has 4 gigabit ports on a round-robin LAG to the first switch giving me near-400MB/sec transfer speeds. And yes, in Btrfs RAID5 and RAID10, I can achieve those read speeds.

ALL of my computers (NAS, HTPC, Desktop, and Laptop) all run Gentoo Linux with dm-crypt on every HDD and SDD for full-disk encryption. The only thing not encrypted is my Windows HDD in my desktop as that has nothing important on there and is strictly for gaming.

FWIW, I have a StarTech 12U rolling rack which houses the "server room" 1U switch, 4U NAS, and an aging Dell PowerEdge 2850 server used as a my in-house web development server plus other backups. Soon this will be retired and will get something more modern and heavy-duty to run a bunch of Qemu/KVM-based virtual machines which give me great success on my laptop and desktop.

User avatar
MinisterofDOOM
Moderator
Posts: 30928
Joined: Wed May 19, 2004 5:51 pm
Car: 1962 Corvair Monza
1961 Corvair Lakewood
1974 Unimog 404
1997 Pathfinder XE
2005 Lincoln LS8
Former:
1995 Q45t
1993 Maxima GXE
1995 Ranger XL 2.3
1984 Coupe DeVille
Location: The middle of nowhere.

Post

How are you liking pfsense? I thought about going that route before discovering VyOS, which I'm still playing with a bit before actually rolling it out to replace my Centurylink modem's router functionality. The Modem has no traffic shaping or QoS settings, so that slow connection becomes even more frustrating.

ArmedAviator
Posts: 526
Joined: Tue Mar 22, 2016 5:28 pm
Car: 2012 Infiniti M37x
Location: SW Ohio

Post

I've never heard of VyOS. I am more of a Linux fan than BSD, as I feel Linux has really stood out in performance and hardware compatibility, and even security as of late. pfSense, however, is a really good product. If I continue to use it, and I foresee that I will, I'll be donating to the developers or maybe buying one of their appliances.

I also forgot to add that I run OpenVPN client on the pfSense box. It routes all traffic on my personal VLAN heading to the outside world through Private Internet Access OpenVPN servers with the exceptions of some specific hosts (Netflix and some email servers mostly). The single-core VIA CPU struggles with more than average web surfing so I'm hoping to upgrade soon.

User avatar
Dattebayo
Posts: 33288
Joined: Sun Aug 25, 2002 10:04 am
Car: 2004 Nissan Frontier Desert Runner
Location: NE DC

Post

Network infrastructure engineer checking in here, with a couple of probably negative sounding, but well meaning observations! :D

Cat6 is literally overkill for a home like yours, and probably useless considering that most pre-term cat5e patch cables can handle 95-100 percent of anything a gigabit switch or connection can throw at it. I personally only use 6 if the customer specs it personally, and I usually talk them out of it unless they have several 300ft or longer runs and it's just cheaper to buy all the same grade cable.

Did you use the special Cat6 RJ45s for the terminations? Again, not trying to get your jimmies up, it's just that alot of people don't know that using standard ends on cat6 makes them the same usage speeds as cat5e.

Also, cat6 made without the plastic "+" separator inside the jacket is practically worthless, hope you got the good stuff.

User avatar
MinisterofDOOM
Moderator
Posts: 30928
Joined: Wed May 19, 2004 5:51 pm
Car: 1962 Corvair Monza
1961 Corvair Lakewood
1974 Unimog 404
1997 Pathfinder XE
2005 Lincoln LS8
Former:
1995 Q45t
1993 Maxima GXE
1995 Ranger XL 2.3
1984 Coupe DeVille
Location: The middle of nowhere.

Post

I was hoping you'd post in this thread.

Yep, I got the good stuff with the plastic separator inside. Makes crimping more of a PITA, so next time I probably will just go CAT5e for that reason alone--except that I have about 900 feet of CAT6 left...heh. And I got cat6 specc'ed RJ45s. I know it's WAY overkill, but the cost difference was almost nothing on Amazon with Prime, so I figured why the Hell not, right? It was $40 for 1000 feet of CAT6, and $9 for 100 RJ45s (and another $10 for 25 keystone terminators).

This is definitely a VERY amateur-grade project, though, so I'm always up for input/advice/suggestions on what I could have done better or what to do next.

User avatar
Dattebayo
Posts: 33288
Joined: Sun Aug 25, 2002 10:04 am
Car: 2004 Nissan Frontier Desert Runner
Location: NE DC

Post

$40 for 1000'? :wtf2:

That must be the cheapest, most Chinese stuff I've ever heard of. Either that, or it fell of the back of a truck somewhere in Jersey. :chuckle:
Good Cat6 generally goes for $250-400 a 1000' box at Graybar, depending on if it's plenum-grade and Brand-name specific or just the cheapo riser PVC stuff. SO, I dunno what you found, but I'm slightly worried.

User avatar
MinisterofDOOM
Moderator
Posts: 30928
Joined: Wed May 19, 2004 5:51 pm
Car: 1962 Corvair Monza
1961 Corvair Lakewood
1974 Unimog 404
1997 Pathfinder XE
2005 Lincoln LS8
Former:
1995 Q45t
1993 Maxima GXE
1995 Ranger XL 2.3
1984 Coupe DeVille
Location: The middle of nowhere.

Post

Eh, it'll serve my needs just fine. Switch is seeing gigabit on those with gig NICs.

This is the cable I bought:
https://www.amazon.com/gp/product/B00QJ ... UTF8&psc=1

It's gone up to $55 now, but that's still not $250.

User avatar
Dattebayo
Posts: 33288
Joined: Sun Aug 25, 2002 10:04 am
Car: 2004 Nissan Frontier Desert Runner
Location: NE DC

Post

Wow. Will test very soon. Don't trust a switch's speed monitor, BTW. You need a real tester.

User avatar
Jesda
Posts: 39644
Joined: Mon May 05, 2003 1:50 pm
Location: STL, DTW
Contact:

Post

People think it's odd that I have a 4 year old cell phone and use crappy free wifi at home to get online, yet I work in IT.
I guess it's like the mechanic who drives a beater. I know how little I can get away with.


That said, if I was in a more permanent living situation, it would be a lot of fun to put something like this together.

ArmedAviator
Posts: 526
Joined: Tue Mar 22, 2016 5:28 pm
Car: 2012 Infiniti M37x
Location: SW Ohio

Post

MinisterofDOOM wrote:Eh, it'll serve my needs just fine. Switch is seeing gigabit on those with gig NICs.

This is the cable I bought:
https://www.amazon.com/gp/product/B00QJ ... UTF8&psc=1

It's gone up to $55 now, but that's still not $250.
It'll work with Gigabit speeds no problem most likely. When I get my own place, I will be using actual Cat-6 spec cable in-wall in an effort to switch to 10Gig-E rapidly as used switches and NICs become more plentiful on eBay.

User avatar
BusyBadger
Posts: 3753
Joined: Thu Apr 28, 2005 3:20 pm
Car: '92 Nissan 240SX
'05 Nissan 350Z
'13 Nissan Juke
Contact:

Post

MinisterofDOOM wrote:And the...er..."rack."
Image
You'd get extra geek points if you would tag those cables with a Dymo label maker! You do get points for your nice network map though! :)

User avatar
Dattebayo
Posts: 33288
Joined: Sun Aug 25, 2002 10:04 am
Car: 2004 Nissan Frontier Desert Runner
Location: NE DC

Post

I have a spare rack laying around if you want it! I removed it from an REI store 2 years ago and it's just collecting dust.

It's an 18U, 24" deep, swing-out locking style rack, all yours! Just drive over and get it. :bigthumb:

User avatar
MinisterofDOOM
Moderator
Posts: 30928
Joined: Wed May 19, 2004 5:51 pm
Car: 1962 Corvair Monza
1961 Corvair Lakewood
1974 Unimog 404
1997 Pathfinder XE
2005 Lincoln LS8
Former:
1995 Q45t
1993 Maxima GXE
1995 Ranger XL 2.3
1984 Coupe DeVille
Location: The middle of nowhere.

Post

Hahaha. Too bad I'm not closer. 18u would probably fit in the crawlspace, even.

User avatar
MinisterofDOOM
Moderator
Posts: 30928
Joined: Wed May 19, 2004 5:51 pm
Car: 1962 Corvair Monza
1961 Corvair Lakewood
1974 Unimog 404
1997 Pathfinder XE
2005 Lincoln LS8
Former:
1995 Q45t
1993 Maxima GXE
1995 Ranger XL 2.3
1984 Coupe DeVille
Location: The middle of nowhere.

Post

Some geeky figures for you:

I've been running a power meter (this one, if anyone's interested) at the combined source for both servers, the switch, and the little LCD monitor at the KVM. Last month showed a consistent average of about 90KWh per week. That's about $9/week in electricity costs. Considering that my most expensive power bill of the year in peak summer with A/C set lower than I need is $130, and a "normal" summer month is $100, that puts the servers alone at 30%-50% of my power bill.

Peak wattage draw was 556.1W. Half a friggin kilowatt.
But, when I consider that I'm replacing the cost of a $300 enterprise grade router/firewall, an enterprise-grade NAS, my own "cloud" storage, and a bunch other stuff, I think I'm still coming out ahead, especially when taking the cost-per-terabyte of cloud storage into consideration. Plus it gives me a great opportunity to learn a lot.

I wonder how the power consumption will scale once I start filling out the unused CPU sockets...

Meanwhile, my beefy gaming rig with triple monitors, upper midrange GPU, two spinning drives, and water cooling (pumps draw more power than fans) peaks at about 230W under highest artificial load (full-system stress test). Much, much more efficient, especially considering the clockspeed per core is more than double that of the servers (4.2 vs 1.7) with the same core count--and that's ignoring the presence of the GTX970 in the desktop.

User avatar
Dattebayo
Posts: 33288
Joined: Sun Aug 25, 2002 10:04 am
Car: 2004 Nissan Frontier Desert Runner
Location: NE DC

Post

I always thought computer power supplies draw the same no matter what load is on them, unless they're adaptable and/or have a special power save setting...


Return to “General Chat”