2020-08-01 20:21:00
The Dell R410 in my homelab has served me very well so far! With a little upgrade of its memory it's run 20 VMs without any hassle. Finding this particular configuration when I did (at a refurbishing company) was a lucky strike: a decent price for a good pair of Xeons and two large disks.
I've been wanting to expand my homelab, to mess around with vMotion, Veeam and other cool stuff. Add in the fact that I'd love to offer "my" students a chance to work with "real" virtualization (using my smaller R410) and you've got me scouring various sources for a somewhat bigger piece of kit. After trying a Troostwijk auction and poking multiple refurbishers I struck gold on the Tweakers.net classified ads!
Pictured above is my new Dell R710, the slightly beefier sister of the R410. It has space for more RAM, for more disk drives and most importantly (for my own sanity): it's a 2U box with larger fans which produces a lot less noise than the R410. The seller even included the original X5550 CPUs seperately.
So! From the get-go I decided to Frankenstein the two boxes, so I could actually put the R410 to use for my students while keeping a bit more performance in my homelab.
Moving that RAID1 set from the R410 to the R710 was an exciting exercise!
I really did not want to loose all of my VMs and homelab; I've put a year into the environment so far! Officially and ideally, I would setup VMware ESXi on the R710 and then migrate the VMs to the new host. There are many methods:
Couldn't I do it even faster? Well sure, but you can't simply move RAID sets between servers! Most importantly: you'll need similar or the same RAID controllers. In a very lucky break, both the R410 and the R710 have the Dell/LSI Perc 6i. So, on a wish and a prayer, I pludged the RAID set and told the receiving Perc 6i to import foreign configuration. And it worked!
After booting ESXi from the SD card, it did not show any of the actual data which was a not-so-fun surprise. Turns out that one manual re-mount of the VMFS file system did the trick! All 24 VMs would boot!
So far she's a beaut! Now, onwards, to prep the R410 for my students.
kilala.nl tags: sysadmin, work,
View or add comments (curr. 0)
2020-07-10 13:45:00
It's official! After passing the theoretical exam in June and completing the practical, virtual classroom assessment this week, I'm now officially CTT+ certified: CompTIA CTT+ Virtual Classroom Trainer Certification.
Many thanks to the people who supported me; you know who you are! 💝
kilala.nl tags: work,
View or add comments (curr. 0)
2020-06-13 22:20:00
One Reddit user suggests that, while my suggested way of working is easier than others, it may also lead to "bricking" of servers: literally rendering them unusable, by applying firmware updates out of order.
Their suggestion is to instead use the SUU (Server Update Utility) ISO image for the server in question, which may be run either from a booted Windows OS, or through the LCM (Life Cycle Manager).
More information about the SUU can be found here at Dell.
Also, if you take a look at Dell's instruction video about using the SUU ISO from the LCM, I think we can all agree that this in fact the easiest method bar none.
EDIT: If it weren't for the fact that the old LCM firmware on the R410 cannot read the SUU files. So you have to use this with Windows or CentOS.
If you want to skip all the blah-blah:
Early in 2019 I purchase a Dell R410, part of Dell's eleventh generation (11G) server line-up from 2010/2011. Since then I've had a lot of fun growing and maintaining my homelab, learning things like Ansible and staying in touch with Linux and Windows administration.
One task system administrators commonly perform, is the upgrading of firmware: the software that's built into hardware to make it work. If you check out the list of available firmware options for the R410, you'll see that quite a lot of that stuff goes into one simple server. Imagine what it's like to maintain all of that stuff for a whole rack, let alone a data center full of those things!
In the case of the R410, support options from Dell are slipping. While many homelabs (and some enterprises) still rock these now-aging servers, the vendor is slowly decreasing their active support.
In my homelab I have tackled only a small number of firmware updates and I'll quickly discuss the best/easiest way to tackle each. In some cases it took me days of trying to figure them out!
Dell's 11G systems (and later) include the Life Cycle Manager (LMC) which makes firmware updates a lot easier. You reboot your server into the USC (Unified System Configurator), launch the updater and pick the desired firmware updates.
Unfortunately, the bad news is that somewhere in 2018 Dell dropped the 11G updates from their "catalogs". You can still use the following steps to make your 11G system check for updates, but it won't find any. You can check the catalogs yourself at https://ftp.dell.com/catalog/. Mind you, based on this forum thread, the Dell ftp/downloads site hasn't been without issues over the years.
"No update is available. Make sure that the Windows(R) catalog and Dell(TM) Update Packages for Windows(R) are used."
There are no more updates for 11G systems available for LCM.
Technically it's possible to make your own internal clone of Dell's software update site. For a large enterprise, that's a great idea actually! Dell's recommended way of setting up a mirror to host updates for your specific systems, is to use the Repository Manager (DRM).
You could also use DRM to create a bootable USB stick that contains the updates you want, so the system can go and update itself, using LCM. Great stuff!
But you're still going to run into the same issue we discussed in the previous paragraph: 11G updates are no longer available through the catalogued repository. You can only get them from the Dell support site, as per below.
So for 11G, forget about DRM. For anything besides the iDRAC, you will need to boot an OS to update your firmware.
Updating the iDRAC integrated management system (if you have it) is the easiest task, assuming that you have the full Enterprise kit with the web GUI.
My R410 runs VMware ESXi which, while it's a Unix, is not supported to run Dell's firmware updates from. Dell support a plethora of Windows versions, a few other OSes and (for the 11G systems) RHEL 5 or 6 (Red Hat Enterprise Linux).
I first wanted to try CentOS 6 (a RHEL 6 derivative), because that's an OS I'm quite comfortable with. I grabbed an ISO for CentOS 6 Live, used dd to chuck it onto a USB stick and booted the OS. Running the BIOS and LCM updates worked fine.
However, the BMC update proved to be quite a mess! In the .BIN package you'll find a rat's nest of shell scripts and binaries which have dependencies not available by default on the CentOS 6 live DVD (like procmail and a bunch of older C libraries). I tried fighting my way through all the errors, manually tweaking the code, but finally decided against it. There has to be an easier way!
Thanks to a forum thread at Dell, I learned that there is in fact an easier way. Instead of fighting with these odd Linux packages, let's go back to good ol' trusted DOS!
I learned that booting FreeDOS from a USB stick on the R410 is problematic. In my case: it's a no-go. So I took FreeDOS 1.3 and burned their Live CD to a literal CD-ROM. Stuck that in the R410's DVD drive and it boots like a charm!
While FreeDOS does not have USB drivers, there is some magic in the underlying boot loaders that will mount any USB drives attached to the system during boot-time. The USB stick I put in the back USB port was made available to me as C:, while the booted CD-ROM was R:.
What do you put on that USB stick? The contents of the PER410*.exe files available from Dell's support site. Each of these is yet another self-extracting ZIP file, containing all the needed tools for the update.
After removing the two iDRAC modules (read below) and getting the correct update (see below also), I followed the instructions from Dell's support team in that forum thread, extracted the ZIP file onto the USB stick, booted FreeDOS and ran "bmcfwud". The system needed a reboot and a second run of bmcfwud. And presto! My BMC was updated!
BMC stands for Baseboard Management Controller. It's Dell's integrated IPMI-based management system, which is literally integrated into the motherboard of the 11G systems. It'll let you do some basic remote management. The most important reason for homelab admins to consider updating BMC is to get version >=1.33 which greatly decreases fan noise.
BMC was superceded by iDRAC (integrated Dell Remote Access Controller), which offers cool features like SSH access, a web GUI and much, much more features! Here's a short discussion about it.
For all intents and purposes iDRAC replaces BMC. If you have an iDRAC installed, the BMC will not be active on your 11G system. The fan noise issues on the R410 should be fixed with any recent version of the iDRAC firmware.
So why did I want to update the BMC firmware?
Because I'm stubborn. =)
Initially, running the updater failed because it said my BMC was at version 2.92. Well, that's impossible!
Turns out, that's because I still had the iDRAC in there! :D I removed both iDRAC daughter cards and tried again.
A downgrade? While I grabbed the most recent BMC update from Dell's site?! No thank you !
So, funny story: Dell's support site for the R410 states that the most recent available version for BMC's firmware is 1.15. The poweredgec.com site for 11G also confirms this. But if you manually search for them, you'll find newer versions:
Apparently my BMC already had 1.54, so it already had the fan updates from 1.33. Guess all the noise that thing was making was "normal". Anyway, grabbing the 1.70 update and running bmcfwud finally had the desired end result.
kilala.nl tags: sysadmin, work,
View or add comments (curr. 0)
2020-01-09 20:42:00
I am very grateful towards my classmates and our teacher Lyda de Groodt, at ballet school La Bayadère in Almere (Facebook). They've always been awesome to our daughter Dana, to my wife (when she still trained there) and now to me.
As they say: "Don't judge a book by its cover". Sure, La Bayadère isn't a big brand-name corporation and no, they don't have some fancy modern studio. But they provide quality: personal guidance, a keen eye, discipline and of course a bit of fun! The students with whom I've trained want to learn proper ballet, but it's clear that we also do it for our own enjoyment. Today, I remarked to S. that "what I really love about this group, are all the smiles and laughs". Our group isn't just focused on rigorous dance, we also connect a bit on a personal level.
I'm really happy to be training with people like T., Q., A. and I.: they never fail to make me feel like I'm twenty years younger again! ^_^
Ballet certainly is different from what you're used to seeing from me, after a sports hiatus of three years and four years of hard-hitting kendo. But I really, really enjoy it. Now that I think of it, it's funny that I haven't started writing about it earlier, seeing how voracious I was about kendo-blogging.
kilala.nl tags: sports, ballet,
View or add comments (curr. 0)
2019-12-27 13:52:00
2020's right around the corner and I've been poking colleagues, urging them to set study-goals for the upcoming year. In Dutch, we have saying equating a lack of progress to deterioration: "Stilstand is de dood" ("Stagnation is death"). I believe that this proverb applies very heavily to work in IT: if you're not keeping up with the times, you're going to get out-dated real quickly.
A colleague asked for suggestions on how to set goals for yourself, to which I replied:
I'd suggest taking into account things like A) where do you want to be in 2-3 years? B) is your team or company lacking particular knowledge or experience? C) do you, or your team, have requirements that you need to fulfill through training? D) do you see any chances that will allow you to quickly up your perceived value?
Basically: train for the job you want, fill any gaps that your team has and make sure you're not dropping any balls.
For me, EX407 fills categories B and C (my current team has little Ansible experience and it will renew my RHCE which will lapse in 1.5 years). The Python for pen-testing course will help me with A (I want to move towards red-teaming and my current coding skills are almost nill).
This year's CySA+ was for category D (it was heavily discounted and I'm pretty sure I could pass it, thus adding a well-regarded cert to my name). Ditto for trying the SANS Work/Study programme, which gets me a heavy discount on a very big-name training and cert.
Finally: just keep a list of things that you want to investigate or work on. Maintain it throughout the year, add new things, remove unwanted things, change priorities. That way you're always set for A) next year's study plans and B) that all-time favorite interview question "Where do you see yourself in two years? What are your short-term development plans?"
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2019-12-09 12:53:00
Another day taken off from work for fun stuff! This time around I went in for yet another CompTIA beta exam, the new CS1-002 CySA+. Like before I sat the exam at my favorite testing center: IT Vitae in Amersfoort. The old Onze Lieve Vrouwe monastery and green surroundings make for a relaxing atmosphere! What was new this time, is that I sat the exam in tandem with my colleague D. She's great company, darn clever and she was looking to get back into the certification-game.
First up, let me point you at a great review of the CS1-002 beta exam, by u/blackvapt on Reddit. And here's the official thread on Reddit, inviting people to take part in the beta.
I will echo everything /u/blackvapt said. The new CySA+ exam is in fact good! The questions are in-depth and technical, without overly focusing on commandline options and flags. In that regard it matches my experience with the PenTest+ exam in 2018: the exam tests for insight and experience in the field of incident response. It's not something you can simply cram books for, you'll need to have experienced many of the situations discussed on the test. The thing is: it's nigh impossible to learn every log format and every OS out there, but if you can intuit the meaning of logs and commands based on your experience, you'll go a long way!
The PBQs (performance based questions) were great! I enjoyed most of them and thought them to be actually fun and a nice multi-layered puzzle. So much better than my experience with the Linux+ exam which only managed to frustrate me with its strict and limited PBQs.
Preparation-wise I'll admit that I took it easy. I was relying mostly on A) my experience from the past 5-10 years, B) the Jason Dion practice exams for CS1-001 on Udemy and C) the Chapple & Seidl book from Sybase. I spent about twenty hours reviewing and researching, over a month's time.
I didn't spend more than $25 on the preparations, as the practice exams were on discount down to $10 and I got the C&S book through Humble Bundle in a large stack of awesome Sybex books. One note about Humble Bundle: I cannot recommend the Packt books or bundles! Skip those. But snag anything you can get from Sybex, NoStarch or O'Reilly!
Regarding the Dion practice tests: I was not passing any of these while preparing as I mentioned earlier this week. It was odd because I felt good on most of the answers I gave to Jason's questions, but I kept missing the passing grade by a fair margin. During the beta exam I felt great about ~85% of the questions, so it's really a crap-shoot on whether I passed the beta or not. :)
If I didn't pass, I wouldn't mind at all! This was a great exam, with solid challenging questions. If I don't make it, I will definitely take the exam again (at full price), now know what to expect.
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2019-12-05 09:31:00
I've got my exam planned for Monday and I'm looking forward to it. I'll mostly treat it as a recon mission, doing it part for fun and part to see if I'd like to take the exam "for real" should I not pass.
I've got a sneaking suspicion I won't pass this time around though (unlike the Linux+, Pentest+ and CFR-310 betas) because my experience keeps tripping me up. Sounds like a #HumbleBrag, I know, sorry :D What I mean is that CompTIA mostly seems targeted at US-based SMB, while my experience comes from EU-based international enterprises. I've been doing a few of Jason Dion's test-exams for the previous version, to get into the right mindset, but I fail a lot of questions because of the aforementioned factors.
Well, let's see how it turns out. For now, I'll just go and have fun with it :)
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2019-11-18 20:59:00
... That's what I told my classmate B. (their ballet blog is here) tonight: "if it were easy, I wouldn't be doing this." That's what I honestly believe: I often do things because they're a challenge. Hence why I kind of live by Bob Ross' quote shown to the left.
Or as Nobel laureate Craig Mello put it: "Ask yourself: “are you having fun?”. And sometimes it’s not fun, but there’s something at the back of your mind maybe saying: “if I can just figure this out”, you know? And when you do, finally do make sense of that thing, man! It’s so much better because it was hard!"
So, what are B., our classmates and myself learning?
Ballet.
I am learning ballet and have been for a few months now. I'm an uncoordinated ditz, struggling with basics, but I'm loving it even when I'm hating it. The hating is short and momentary, the loving is something that sticks.
kilala.nl tags: sports, work, ballet,
View or add comments (curr. 0)
All content, with exception of "borrowed" blogpost images, or unless otherwise indicated, is copyright of Tess Sluijter. The character Kilala the cat-demon is copyright of Rumiko Takahashi and used here without permission.
