edruder.com

Defense in Depth for Real-Time Rails

2026-02-01T00:00:00-08:00

In the previous post, I fixed a race condition between Turbo and ActionCable that caused connected clients to diverge during rapid interactions. The fix was straightforward: bypass Turbo’s form submission lifecycle for actions that broadcast via ActionCable, and return head :no_content instead of redirecting. One update path instead of two. No more interference.

That fixed the loudest problem. But the investigation didn’t stop there.

As I traced the behavior through every layer–server concurrency, broadcast delivery, WebSocket lifecycle, client-side state management–I found issues at each one. A read-then-write race in the mutation method that could silently merge concurrent adjustments. A polling interval tuned for low database load rather than real-time responsiveness. A WebSocket disconnection handler that did literally nothing, silently dropping any broadcasts missed during a network blip. And fixing the Turbo lifecycle issue inadvertently resolved a separate class of bugs–phantom sound effects–that I hadn’t fully diagnosed yet.

No single fix would have been sufficient. The Turbo race from the previous post was the most visible failure mode, but each of these quieter issues could independently cause clients to show stale state under the right conditions. The fixes work as layers, each one addressing a different failure mode.

The specific bugs were in a poker timer, but the patterns are general to any Rails app doing real-time sync with ActionCable. This post is less of a debugging story and more of a hardening guide–four fixes, walking from server to client, that made the feature reliable under production conditions.

Pessimistic Locking

The timer’s adjust_time! method had a concurrency bug hiding in plain sight. It read the current state, computed a new value, and wrote it back–without holding a database lock during the read:

def adjust_time!(minutes)
  elapsed_seconds = Time.current - started_at
  current_computed_time = [current_time_remaining - elapsed_seconds, 0].max
  new_time = current_computed_time + (minutes * 60)

  update!(current_time_remaining: new_time, started_at: Time.current)
  broadcast_time_adjusted
end

The timer record is loaded by a before_action at the start of the request (@timer = @event.timer). Each Puma thread gets its own copy of the record with whatever values were in the database at that moment.

If two requests arrive close together, both threads read the same starting state, both compute the same result, and both write it–producing one effective adjustment from two requests.

This is a time-of-check-to-time-of-use race, sometimes called TOCTOU: the state you checked is no longer the state you’re acting on by the time your write lands. This race exists regardless of your database–PostgreSQL, MySQL, and SQLite are all vulnerable when the application reads state in one step and writes in another without holding a lock.

My app uses SQLite, whose single-writer serialization prevents the writes themselves from interleaving, but that doesn’t help–the damage is done in the reads, which happen before either thread tries to write.

In development, this race never triggered. With a local server responding in single-digit milliseconds, Turbo’s button-disable behavior serialized the requests–each response arrived and re-enabled the button before the next click could fire. Requests never overlapped.

Here’s the irony: the data-turbo="false" fix from the previous post actually made this race more likely, not less. With Turbo bypassed, the browser sends standard form submissions with no Navigator lifecycle managing them. Each click fires an independent HTTP request immediately, with no button-disable serialization.

On a connection with 50-100ms of latency, three rapid clicks easily produce overlapping requests. The Turbo fix solved the client-side race but opened the door to a server-side one. Defense in depth means fixing both.

The fix is pessimistic locking–acquire an exclusive row lock before reading, so no other thread can read stale state. Active Record’s with_lock method wraps the block in a transaction and acquires an exclusive row lock–equivalent to transaction { lock!; ... }:

def adjust_time!(minutes)
  with_lock do
    reload  # re-read current state inside the lock
    elapsed_seconds = Time.current - started_at
    current_computed_time = [current_time_remaining - elapsed_seconds, 0].max
    new_time = current_computed_time + (minutes * 60)

    update!(current_time_remaining: new_time, started_at: Time.current)
  end
  broadcast_time_adjusted
  true
end

Two details matter here.

First, reload inside the lock is critical. Without it, with_lock acquires the exclusive lock but the method still operates on the stale in-memory attributes loaded at the start of the request. The lock alone doesn’t refresh them.

Second, broadcast_time_adjusted is outside the with_lock block–broadcasting doesn’t need the lock, and keeping the critical section small avoids holding it longer than necessary.

This pattern applies to any mutation method that reads current state, computes a new value, and writes it back. It’s distinct from idempotency–each timer adjustment is intentionally non-idempotent (clicking “-5 minutes” three times should subtract 15 minutes), so you can’t solve this by making the operation safe to retry. You need the lock to ensure each read sees the result of the previous write.

Tuning Solid Cable for Real-Time Use

Solid Cable is the database-backed ActionCable adapter that ships as the default in Rails 8. Instead of Redis pub/sub, broadcasts are inserted as rows into a cable messages table, and a listener thread polls for new messages at a configurable interval. That interval determines the maximum delay between a broadcast being written and a client receiving it.

Mine was set to 0.5 seconds–a reasonable default that keeps database load low. For most ActionCable use cases (chat messages, notifications, dashboard updates), half a second of delivery latency is fine. For a real-time feature where users expect instant feedback from rapid interactions, it’s not.

The polling interval also affects batching. When multiple broadcasts are written within a single poll cycle, they’re all picked up and delivered together. This isn’t a problem by itself–the client processes them sequentially in order.

But it widens the timing window for the kind of race conditions I described above. A broadcast that would have been delivered immediately with push-based pub/sub instead sits in the database for up to 500 milliseconds, during which other requests are being processed, other broadcasts are being written, and the client’s state may be shifting underneath.

This wasn’t the cause of the Turbo race condition from the previous post–that bug exists regardless of your ActionCable adapter. But the polling interval amplified it. Narrower delivery windows mean less opportunity for timing- dependent failures.

The fix is a one-line config change:

# config/cable.yml
production:
  adapter: solid_cable
  polling_interval: 0.1.seconds  # was 0.5.seconds

The trade-off is straightforward: 5x more frequent polling means 5x more database reads. For a small-scale app on SQLite, this is negligible. At scale, you’d want to benchmark it.

But the general principle holds: adapter defaults are tuned for broad compatibility, not for your specific latency requirements. If you’re building a feature that depends on fast broadcast delivery, check what your adapter is actually doing.

(This only applies to polling-based adapters like Solid Cable. If you’re using Redis-backed ActionCable, pub/sub delivers messages immediately and there’s no polling interval to tune.)

The Phantom Sounds

After fixing the timer sync issue, a separate problem surfaced: brief, phantom sound effects during rapid adjustments. The timer has an audio alert that plays when the blind level advances–a double beep. This sound was firing during interval adjustments on timers that weren’t even running.

Users would hear the start of a beep, cut short almost immediately, as if something triggered the audio and then killed it. This one had me stumped for a while–I initially suspected some kind of audio race condition or localStorage corruption.

Turns out, two bugs were interacting to produce it.

The first bug was spurious broadcasts. The timer is implemented as a Rails engine mounted in a host application. The host app had a before_action that called sync_with_client_state! on every timer request–including adjustment actions.

The engine’s adjustment methods explicitly avoided this call. The code comments said as much: “We intentionally do NOT call sync_with_client_state! here.” The engine treats the client as the source of truth for running timers–the client advances blind levels autonomously, and the server catches up at defined sync points. By adding sync_with_client_state! to a blanket before_action, the host app overrode that design.

The consequence: when sync_with_client_state! ran during an adjustment request, it compared the server’s database state to the computed client state. For a running timer, the server is typically behind–that’s by design. So the sync method “caught up” by advancing blind levels and broadcasting blinds_increase to all clients. These broadcasts were spurious. The blinds hadn’t actually advanced; the server was just reconciling a gap that the engine was designed to tolerate.

The fix was a one-line deletion: remove sync_with_client_state! from the host’s before_action. The engine already calls it at the right times–on page load, on channel subscription, and via a periodic sync job.

The lesson generalizes: when a library comments “we intentionally do NOT do X here,” that’s a design constraint, not a suggestion. Understand the reasoning before overriding it.

The second bug was stale Stimulus controller state during Turbo page replacement. When a spurious blinds_increase broadcast arrived at a client, it triggered the audio alert path. But Turbo was simultaneously replacing the page (following the redirect from the adjustment action).

During page replacement, there’s a window where the old Stimulus controller is still connected–it still has its ActionCable subscription, it still receives broadcasts, and it still acts on them. The old controller received the blinds_increase broadcast and started playing the alert sound. Then Turbo finished the page swap, disconnect() fired on the old controller, and the audio was paused mid-play–producing the abbreviated beep the user heard.

The audio manager compounded the issue–its teardown method was designed to preserve in-flight sounds during navigation, which meant it actively started audio playback during Turbo page replacement.

This bug was already fixed by the time I diagnosed it! The data-turbo="false" change from the previous post eliminated Turbo page replacement for adjustment actions entirely. No page replacement means no Stimulus disconnect/reconnect churn, no window where an old controller receives broadcasts, and no stale audio state. The phantom sounds disappeared as a side effect of fixing the sync race condition.

That’s worth noting as its own lesson: when you fix a lifecycle bug, look for other symptoms that share the same root cause. I investigated the phantom sounds as a separate issue, with separate hypotheses about audio race conditions and localStorage corruption. They turned out to be downstream of the same Turbo page replacement cycle that caused the timer divergence. The fix was the same because the cause was the same.

WebSocket Reconnect Recovery

ActionCable subscriptions provide connected and disconnected callbacks on the client side. The disconnected callback fires when the WebSocket connection drops–network blips, server restarts, mobile devices switching between WiFi and cellular, laptop sleep/wake cycles.

My timer controller’s handleDisconnected() implementation was:

handleDisconnected() {
  // (empty)
}

During a disconnection, the server keeps broadcasting. With Solid Cable, those broadcasts are written to the database as usual. But without a connected WebSocket on the other end, they can’t be delivered. The messages sit in the cable table until they’re trimmed, and the disconnected client never sees them.

The recovery mechanism already existed on the server side. When the client reconnects, ActionCable re-subscribes to the channel, which triggers the server-side subscribed callback. Mine already called transmit_timer_state, which reads the current timer state and sends it as a point-to-point message to the reconnecting client.

The server was offering the client a way to catch up. The client just wasn’t taking it.

The fix is small. On disconnect, flag that I may have missed broadcasts. On reconnect, clear the flag and force-accept the next server state–bypassing the staleness checks that would normally compare it against local state that may no longer be current:

handleDisconnected() {
  this.sync.missedBroadcasts = true
}

handleConnected() {
  if (this.sync.missedBroadcasts) {
    this.sync.missedBroadcasts = false
    this.sync.initialized = false  // force-accept next server update
  }
}

The re-subscription triggers transmit_timer_state on the server, which sends the current state as a point-to-point message. By resetting initialized, the client treats that message as an initial sync rather than an incremental update, accepting it unconditionally instead of comparing it against stale local state.

Brief WebSocket disconnections are more common in production than you might expect. WiFi access points hand off connections. Mobile networks switch cells. Laptops close and reopen. Server deploys cycle processes.

In development, the WebSocket connects to localhost and effectively never drops. In production, disconnections are routine–and if your disconnected callback is empty, every one of them is a silent opportunity for client state to drift.

The general pattern: if your ActionCable subscription manages client-side state, implement the disconnected callback. At minimum, flag that the client may have missed updates. If the server already sends state on subscription (a common pattern), you may not need additional logic–just ensure the client is ready to accept that state as a full resync rather than an incremental update.

This is the catch-all layer. The pessimistic locking ensures the server state is correct. The Solid Cable tuning ensures broadcasts are delivered promptly. The Turbo fix ensures the client processes them without interference. But even if all of those are working, a network blip can still cause a client to miss a broadcast. This layer ensures it recovers.

The Compound Fix

Here’s what I ended up with across both posts–five fixes at four layers of the stack:

Layer	Problem	Fix
Turbo form lifecycle	Navigator abort race on rapid clicks	`data-turbo="false"` bypasses Turbo
Server concurrency	Read-then-write race in mutation methods	Pessimistic locking with `with_lock` + `reload`
Broadcast delivery	500ms polling widens timing windows	Reduced to 100ms
WebSocket resilience	No recovery from missed broadcasts	`handleDisconnected` / `handleConnected`

The phantom sounds didn’t need their own fix–they were downstream of the Turbo lifecycle and spurious broadcast issues that the other fixes addressed.

No single row in that table would have been sufficient on its own. The Turbo fix eliminated the most visible race but exposed the server-side concurrency bug. The pessimistic locking ensured correct server state but couldn’t help if a broadcast was missed due to a network blip. The Solid Cable tuning reduced delivery latency but didn’t prevent the race. The WebSocket reconnect recovery catches everything else–but only after the fact.

Each layer catches failures that the others miss. That’s defense in depth applied to application code: you don’t rely on one control, because no single layer can account for every failure mode. The server ensures correctness. The transport ensures timely delivery. The client ensures recovery.

Every one of these bugs was invisible in development. Localhost round-trips in single-digit milliseconds eliminate the timing windows where races happen. WebSocket connections to localhost don’t drop. Solid Cable polling latency doesn’t matter when the broadcast and the response arrive almost simultaneously.

Production–real devices, real networks, real latency–is where these layers get tested. If you can’t reproduce something locally but users are reporting it, the problem is probably in a timing window that your development environment is too fast to open.

If any of this sounds familiar, I’d love to hear about it–leave a comment!

The Turbo + ActionCable Trap: When Your Real-Time Rails Feature Fights Itself

2026-01-31T00:00:00-08:00

Two phones, same app, same page containing a timer. One shows 4 minutes on the timer. The other shows 9. The server says 4. Refresh the lagging phone and it snaps to 4 instantly–which makes it maddening to debug, because the bug fixes itself the moment you look at it.

We built a shared timer for our weekly poker game—a Rails 8 app using Turbo for form handling and ActionCable for real-time sync across devices. It worked perfectly in development. It worked perfectly in production, too, right up until someone started clicking the “minus 5 minutes” button a few times in quick succession. Then the clients would diverge.

The pattern was consistent in some ways but not others: one device would fall behind by exactly one adjustment–usually the device that was clicked, but sometimes the passive observer lagged. Any interaction on the lagging device would immediately fix it. And it only happened with rapid clicks, never with deliberate single clicks spaced a few seconds apart.

Here’s what made it hard: Turbo and ActionCable were both doing the right thing. The problem was that they were both doing it at the same time, on the same state, and their page lifecycles were silently interfering with each other.

It took two rounds of investigation and fixes across multiple layers to fully resolve–because the first fix addressed a real problem but exposed a deeper one underneath.

The Architecture That Seems Right

The pattern we used is one you’ll find in a lot of Rails apps that combine Turbo with real-time features. It looks like this:

User clicks a button (say, “adjust timer down by 5 minutes”)
Turbo submits the form as a background HTTP request (no page reload)
The server processes the mutation and broadcasts the new state via ActionCable
All connected clients receive the broadcast and update their UI
The server responds with a redirect back to the edit page
Turbo follows the redirect and re-renders the page with fresh server data

This is textbook Rails. Turbo handles forms. ActionCable handles real-time sync. The controller action looks like what you’d expect:

# In the controller's update action
when "adjust_time"
  minutes = params[:minutes].to_i
  if @timer.adjust_time!(minutes)
    redirect_to edit_path
  else
    redirect_to edit_path, alert: "Cannot adjust time"
  end

And inside adjust_time!, the model broadcasts after persisting the change:

def adjust_time!(minutes)
  # ... compute new time, update the record ...
  broadcast_time_adjusted  # ActionCable broadcast to all subscribers
  true
end

Every connected client has a Stimulus controller subscribed to the ActionCable channel. When a broadcast arrives, it updates the display. Clean separation of concerns. Each piece doing its job.

But notice step 5 and step 6 above. After the server broadcasts the new state to all clients via ActionCable, it also redirects the submitting client back to the page–which Turbo follows, replacing the page body with a fresh render from the server.

The clicking device gets the new state twice: once from the ActionCable broadcast, and again from the Turbo redirect and re-render. In normal operation, this is invisible–both paths deliver the same data, and whichever arrives second is effectively a no-op.

This worked flawlessly in development. Two browser windows side by side, rapid clicks, perfect sync every time.

The reason was simple: with a local server, request round-trips complete in single-digit milliseconds. The timing window where anything could go wrong was vanishingly small. It wasn’t until the app was running in production–real devices on real networks with real latency–that the window opened wide enough for the race to happen.

What Went Wrong

In production, the bug showed up like this: a user would click the adjustment button three times in quick succession. One device would end up showing the correct value–reflecting all three adjustments. The other would show a value that was off by exactly one adjustment, as if it had only received two of the three broadcasts.

A few things made this hard to pin down:

It was intermittent. Slow, deliberate clicks never triggered it. You had to click fast enough that requests overlapped or came in within a few hundred milliseconds of each other.

Either device could be the one that lagged. Usually it was the device that was clicked, but sometimes the passive observer fell behind instead. There was no consistent “active client is right, passive client is wrong” pattern–which made it harder to form a hypothesis.

It self-healed. Any action on the lagging device–clicking a button, navigating away and back, even just refreshing–would snap it to the correct value. The server always had the right state. The problem was purely in how that state reached (or failed to reach) the client.

The server was correct. Every adjustment was processed and persisted. The database always reflected all three clicks. This wasn’t a lost-update problem on the server side–it was a delivery or acceptance problem on the client side.

If you’ve built real-time features with Turbo and ActionCable, you might recognize this pattern. My first instinct was to suspect ActionCable itself–maybe a broadcast was dropped, maybe Solid Cable’s polling missed something. I went down that road. It wasn’t the answer.

The First Fix (That Wasn’t Enough)

The dual update path I described above–ActionCable broadcast plus Turbo redirect/re-render–was the obvious place to start. If the two paths could interfere with each other, maybe eliminating one would fix the problem.

Looking at the controller action, every successful adjustment ended with redirect_to edit_path. That meant Turbo was following the redirect, fetching a fresh page, and replacing the entire page body. During that replacement, the Stimulus controller that managed the timer display would disconnect (tearing down its ActionCable subscription) and a new one would connect (creating a fresh subscription).

There was a window–brief, but real–where the client had no ActionCable subscription at all.

If a broadcast arrived during that window, it would be lost. The client would never receive it. Meanwhile, the Turbo re-render would show whatever the server’s state was at that moment–giving the clicking device a second chance to get the right answer even if it missed a broadcast.

This could explain why sometimes the passive device lagged (it missed a broadcast and had no re-render fallback), and it could also explain why sometimes the clicking device lagged (the re-render showed an intermediate state while a later broadcast was lost during the subscription gap).

This seemed like a plausible explanation. The fix was straightforward: replace redirect_to edit_path with head :no_content for adjustment actions. Since ActionCable was already broadcasting the new state to all clients, the redirect was redundant–its only purpose was to re-render the page, which the broadcast already handled.

when "adjust_time"
  minutes = params[:minutes].to_i
  if @timer.adjust_time!(minutes)
    head :no_content  # was: redirect_to edit_path
  else
    redirect_to edit_path, alert: "Cannot adjust time"
  end

I deployed the fix and tested it. Still broken!

The redirect was a real problem–eliminating it removed the Stimulus disconnect/reconnect churn and the broadcast-loss window that came with it. But it wasn’t the whole story.

With the redirect gone, both clients were now relying entirely on ActionCable for updates. No more asymmetric recovery path. And yet one client still fell behind by exactly one adjustment during rapid clicks. Something else was going on.

The Real Problem: Turbo’s Hidden Page Lifecycle

With the redirect eliminated, I went deeper. If both clients were now getting updates exclusively through ActionCable, and the server was sending all three broadcasts, where was one getting lost?

Dead end: client-side validation logic. The Stimulus controller had staleness checks and time-direction validation on incoming broadcasts. I traced every code path with concrete values for three sequential time_adjusted broadcasts. Every one passed every guard. The JavaScript was not rejecting valid broadcasts.

Dead end: Solid Cable delivery. I traced the full broadcast pipeline end-to-end–from ActionCable.server.broadcast through Solid Cable’s SQLite message table, the polling listener thread, the event loop thread pool, ActionCable’s worker pool, and finally the WebSocket write. There is no mechanism in that pipeline for silent message loss. If a message is inserted into the cable database, it will be delivered to all subscribers. Solid Cable was not the problem.

That left Turbo. Even with head :no_content instead of a redirect, Turbo still manages the form submission lifecycle–and its behavior during rapid submissions is where the problem lives.

Here’s what Turbo’s Navigator does when a form is submitted:

submitForm(form, submitter) {
    this.stop();
    this.formSubmission = new FormSubmission(this, form, submitter, true);
    this.formSubmission.start();
}

That this.stop() call is the key. If a previous form submission is still in-flight when a new one starts, Turbo aborts it. The previous fetch request is cancelled via its AbortController.

This is sensible behavior for navigation–you don’t want two competing page loads. But for rapid-fire mutations that broadcast via ActionCable, the consequences are subtle.

Turbo also disables the submit button when a form submission begins, and re-enables it when the response arrives. For a fast local server returning 204 No Content in a few milliseconds, the button is disabled for an imperceptibly short time–rapid clicks all get through.

But for a production server with real network latency, the timing changes. Clicks might land while the button is disabled (and be silently swallowed), or they might land in a narrow window where an aborted submission’s cleanup asynchronously re-enables the button just after the new submission disabled it.

The result: depending on network conditions, some combination of three rapid clicks produces only two server requests, or produces three server requests where one is aborted client-side. The server processes what it receives and broadcasts for each one. But the number of clicks, the number of server-side mutations, and the number of broadcasts received by each client can all differ–by exactly one–depending on timing.

The Fix

The solution follows directly from the diagnosis: if Turbo’s form submission lifecycle is interfering with ActionCable-driven updates, take Turbo out of the equation for these specific actions.

Step 1: Bypass Turbo for broadcast-driven actions.

Add data-turbo="false" to any form that triggers a server-side mutation which is broadcast to clients via ActionCable:

<%= button_to timer_path(@timer),
      method: :patch,
      params: { action_type: "adjust_time", minutes: -5 },
      data: { turbo: false } do %>
  <span>-5 min</span>
<% end %>

With data-turbo="false", the form submits as a standard browser request–no Navigator, no this.stop(), no AbortController, no button disable/enable dance. Each click fires an independent HTTP request. The browser ignores the 204 No Content response (no navigation, no page replacement), and the UI update comes entirely through ActionCable. Every client gets its update through the same path.

Step 2: Return head :no_content instead of redirecting.

This was my first fix, and it’s still necessary. Without Turbo managing the form, a redirect would cause a full page reload (standard browser behavior for a non-Turbo form getting a 302). Returning 204 No Content tells the browser there’s nothing to do with the response–the ActionCable broadcast handles the UI update.

when "adjust_time"
  minutes = params[:minutes].to_i
  if @timer.adjust_time!(minutes)
    head :no_content
  else
    redirect_to edit_path, alert: "Cannot adjust time"
  end

Note that the error case still redirects–if the adjustment fails, I want to show the user an alert, and there’s no race condition concern because no broadcast was sent.

What this changes architecturally:

Before the fix, adjustment actions had two update paths: the ActionCable broadcast (for all clients) and the Turbo redirect/re-render (for the clicking client only). These paths raced against each other and against Turbo’s Navigator lifecycle.

After the fix, there is exactly one update path: the ActionCable broadcast. Every client–whether it clicked the button or not–receives the update the same way. No asymmetry, no race, no timing dependence.

What I Took Away From This

It’s possible that there’s a better way to fix this kind of problem. The fix described here does seem to work, and work well. But it’s possible that there’s a better solution–maybe one that doesn’t entail disabling Turbo in certain contexts. If you think of a better solution, I’d love to hear about it!

The specific bug was in a poker timer, but the underlying pattern is general. Any time you have a Rails action that does both of these things:

Broadcasts state via ActionCable (or any WebSocket/real-time channel)
Responds with a Turbo-managed redirect or page update

…you might have two competing update paths for the same state change. In the normal case–one user, one action, low latency–they deliver the same data and you never notice the duplication. But under pressure (rapid actions, multiple clients, production latency), they interfere with each other.

Here’s how I think about the interference now:

Turbo owns the page lifecycle. When Turbo manages a form submission, it controls the fetch request, the response handling, the page replacement, and the Stimulus controller lifecycle. If it decides to abort a request, disable a button, or replace the page body, those decisions cascade through everything else on the page–including your ActionCable subscriptions and Stimulus state.

ActionCable assumes a stable connection. Broadcasts are fire-and-forget from the server’s perspective. If the client’s subscription is torn down (because Turbo is replacing the page) or the client’s JavaScript is busy processing a Turbo lifecycle event, the broadcast has no way to retry or confirm delivery.

The combination creates asymmetry. The device that triggered the action gets special treatment from Turbo (re-render, fresh server state, Navigator lifecycle management). Every other device gets only the ActionCable broadcast. When Turbo’s lifecycle interferes with broadcast delivery or acceptance, the devices diverge–and they diverge in ways that depend on network timing, making the bug intermittent and hard to reproduce in development.

The fix isn’t “use ActionCable better” or “use Turbo better.” It’s recognizing that for actions whose results are broadcast to all clients in real time, Turbo should not be managing the form submission at all. Let the browser handle the HTTP request. Let ActionCable handle the state update. One path, one source of truth, no interference.

Signs You Might Have This Bug

If your app matches several of these, you’re probably vulnerable to this class of problem:

A Turbo-submitted form triggers an ActionCable broadcast. This is the core pattern. The form response and the broadcast are two delivery paths for the same state change.
Multiple clients observe the same state. If only the submitting user cares about the result, the dual update paths are harmless–there’s no other client to diverge from. The bug requires at least two clients.
Users perform rapid or repeated actions. Single actions with seconds between them are unlikely to trigger the race. The timing window opens when actions overlap or arrive within a few hundred milliseconds of each other.
Your controller redirects after a successful mutation. The redirect-and-rerender cycle is what causes Stimulus disconnect/reconnect and the ActionCable subscription gap. head :no_content is necessary but not sufficient on its own–Turbo’s Navigator still manages the form submission.
You’re using Solid Cable or another polling-based ActionCable adapter. Polling introduces delivery latency that widens the race window. The default Solid Cable polling interval of 0.5 seconds means broadcasts can queue for up to half a second before delivery. This isn’t the cause of the bug–the core vulnerability exists regardless of your ActionCable adapter–but polling makes it more likely to manifest. Even with Redis-backed ActionCable, the race window exists; it’s just narrower.
Your production environment has meaningful network latency. This is what makes the bug invisible in development. Local round-trips in single-digit milliseconds don’t give the race enough time to trigger.

If you recognize this pattern in your app, the fix is the same regardless of what your feature does: add data-turbo="false" to forms whose successful actions are broadcast via ActionCable, and return head :no_content instead of redirecting.

Coming Up

Bypassing Turbo and eliminating the redirect fixed the primary race condition. But during the investigation, I found several other issues that contributed to the unreliability–a read-then-write race in the mutation method, a no-op WebSocket disconnection handler that silently dropped broadcasts, and a Solid Cable polling interval that was tuned for low DB load rather than real-time responsiveness. I also discovered that fixing the Turbo lifecycle issue inadvertently resolved a separate class of bugs–phantom sound effects triggered by stale Stimulus controller state during page replacement–that I hadn’t fully diagnosed yet.

In the next post, I’ll cover the defense-in-depth fixes that made the real-time feature truly robust: pessimistic locking, WebSocket reconnect recovery, and Solid Cable tuning.

Today I Learned How to Keep Christmas Tree Lights Working Forever

2024-01-24T00:00:00-08:00

I have been decorating Christmas trees with incandescent mini-lights almost every year for many years. (LED lights are different.) Over that time, I’ve bought, maintained, and thrown away dozens of strands of lights. I’m cheap, so I always try to repair a strand when some lights go out, and I generally keep strands working for years. Eventually though, each strand gets to a point where I can’t get it working in a reasonable time, and I throw it away and buy a new one.

That is, until this year, when I learned how to keep Christmas tree lights going virtually forever!

Get a LightKeeper Pro

A while ago, I discovered a great tool to keep tree lights working when a light bulb or two goes out–the LightKeeper Pro (LKP)! It’s a super-useful tool–I think that everyone who lights a Christmas tree every year should have one. It has several functions, but the one I use the most is the quick fix–you plug your lights into the gun-shaped tool, pull the trigger a few times, and the section of your strand that was dark magically lights up!

It’s worth mentioning that most Christmas tree strands are composed of several sections of lights that are connected together to make a longer string of lights. For example, a 150-light string is actually made up of three 50-light sections connected end-to-end. (There are strings of lights that are composed of 35-light sections, but I don’t have any of those. These suggestions still apply.) When a bulb in one of the sections blows out, it will only affect its own section of lights. That’s why most of a long string of lights will remain working when one section goes dark. When I say “strand”, I mean one of these 50-(or 35-)light sections of a longer string of lights.

The LightKeeper Pro Isn’t Enough

Using the LKP quick-fix alone doesn’t keep your lights going forever though. I had some tree lights that had a bulb or two burn out every year or two and I’d use the LKP to get them working again. But then the entire strand of light bulbs blew out–every single bulb was blackened on the inside!

This year, as usual, the LKP fixed some of my lights before I put them up. After I fixed them and put them up, we finished decorating the tree. The very next day, a strand went out and the LKP didn’t help–all the bulbs were blown. Another strand completely burned out the next day! While I’d seen this happen before once or twice, this was the first time it happened to me after the tree was completely decorated. We decided to live with the burned-out strands for the duration of the season–removing the lights and ornaments, fixing the lights, then putting up the lights and ornaments again would have been too much work.

As I chewed on this frustration, I decided to figure out what caused this, and it didn’t take long to find out.

Replace All Burned Out Bulbs

The short answer is that every season, before you put up your lights, replace all the burned out bulbs. It turns out that as long as you start each season with all of your mini light bulbs working, the LKP quick fix will keep you going for the season.

You usually get a few replacement bulbs with each new strand. When you run through those, you can buy mini light bulbs in bulk online or at many hardware stores. (The number of bulbs in a strand determines the voltage of the bulbs you need to buy–2.5V for 50-bulb, 3.5V for 35-bulb strands–make sure to get the right ones.)

How This Works

If you just want to keep your lights working, you can stop reading. If you’re curious why these suggestions work, here are the details.

How Does The LightKeeper Pro Work?

It’s not magic, of course. I learned that each bulb in a strand has a little shunt built into it. The shunt’s job is to keep the rest of the bulbs in the strand lit even when its bulb burns out. Usually the shunt just works–when a bulb burns out, the shunt kicks in and the rest of the bulbs continue to work. Fairly frequently though, a bulb’s shunt doesn’t work right away, and the entire strand goes dark. That’s where the LKP comes in–plugging the strand into the tool and pulling the trigger generates a jolt of electricity that causes the reluctant shunts to start working, and the strand lights up!

Why Replace All The Dead Bulbs?

If you just keep clicking the LKP to get your strands working every year, more and more bulbs in each strand will be burnt out. Not very noticeable, but each strand is getting closer and closer to catastrophe. Here’s what’s happening.

When a strand is new and every bulb is lit, there’s a certain amount of electric current running through the strand. When one bulb burns out and its shunt kicks in, the same amount of current is going through the strand, but it’s lighting one fewer bulb. This causes each of the remaining bulbs to burn a little brighter. When another bulb burns out, the remaining bulbs burn a little more brightly. When a few more bulbs (5? or 6? maybe more?) have blown out, the remaining bulbs are running as hot as they can. The next bulb to die sets off a cascade of blown bulbs until every one is burned out.

A strand of lights can withstand a few dead bulbs, so just start the season off with no dead bulbs and your season will be merry!

The LightKeeper Pro Is Even Better!

There are a few more ways that your strands can go dark and the LKP can help you figure out the problem:

Every string of lights has a little fuse in the plug which can blow–the LKP has a fuse tester.
Sometimes, the wires at the base of a bulb can break. The LKP has a voltage detector that can tell you where the bad bulb is.
The LKP has a bulb tester, too.

References

Easy Property files for Bash scripts

2020-09-13T00:00:00-07:00

Recently, I wanted to have a setting/preference to control an aspect of a Bash script. I considered using an environment variable, but I wanted the preference to be able to be different based on the directory that the script is run from—the “local” directory’s setting would override the “global” setting.

I’ve used a hierarchy of YAML files from Java—but YAML is overkill and I just needed two levels of hierarchy. A simple “properties file”, with settings in the local file overriding the settings in the global file in the home directory, works. A properties file is a text file, usually with a .properties extension, with one property per line, and each line looking like this: property=value.

Searching online, I found a simple sed command that nicely plucks specific properties from a file. With that, I wrote a simple Bash function that returns the value of a property from a properties file, looking first in the properties file in the directory where the script was invoked, then in the properties file in the home directory.

Here’s the function:

prop() {
  name=$1

  local_properties="$(pwd)"/"$properties_filename"
  if [[ -f "$local_properties" ]]; then
    value=$(sed -En "s/^$name=([^\n]+)$/\1/p" "$local_properties")
  fi

  if [[ -z "$value" ]]; then
    global_properties=~/"$properties_filename"
    if [[ -f "$global_properties" ]]; then
      value=$(sed -En "s/^$name=([^\n]+)$/\1/p" "$global_properties")
    fi
  fi

  echo "$value"
}

The function expects the variable properties_filename to be defined.

It’s invoked like this:

if [[ "$(prop 'allow_something')" == "true" ]]; then
  # do something
fi

Note:

This is on macOS, which has slightly different syntax for the sed options than is described in the original article (macOS: -En vs. Linux: -rn).
The function interprets an “empty” value as “no property is defined”, so you probably don’t want to define a property without a value (i.e., property=).
I usually make my preferences files invisible.
I usually don’t commit my preferences files to source control.

Using a Bootstrap DateTime Picker in Simple Form and Rails 5.1

2018-01-27T00:00:00-08:00

I’m using Bootstrap to make the pages of my Rails app look halfway decent–I’m a programmer who’s creatively challenged. (I can already feel the scorn from my designer friends, but hey–I actually like how Bootstrap looks!) I’m also using Simple Form to clean up the HTML forms in my app.

In one of my forms, the user needs to choose a date & time, and the default way to enter a date & time in Rails is horrendously ugly, even to my soulless, dead eyes. A little Googling for ‘bootstrap datetime picker rails’ landed me on the bootstrap3-datetimepicker-rails gem page. This gem just wraps the bootstrap-datetime-picker, whose minimal setup looked perfectly suitable to me. Piece of cake, I thought!

It was straightforward to install and configure the gem using the instructions on its home page. The hard part was trying to figure out how to use the picker in my Simple Form form.

The bootstrap-datetime-picker home page describes the markup that renders the picker. From the top of that page, this is what I wanted Simple Form to output:

<div class="form-group">
  <div class="input-group date" id="datetimepicker1">
    <input type="text" class="form-control" />
    <span class="input-group-addon">
      <span class="glyphicon glyphicon-calendar"></span>
    </span>
  </div>
</div>

(There is some markup wrapping this, but this is the part that is important.)

I read the (very complete!) Simple Form documentation, but I didn’t see how to specify a stock Simple Form field that would result in something resembling the markup above. I read several of the Simple Form wiki pages about building custom input components, and I decided that that’s what I needed to create to get the markup that would render the datetime picker properly.

Unfortunately, that documentation didn’t get me to the point where I knew how to write the custom component, but with that and several other examples I found online, my mental picture of what I needed to do became clearer. I didn’t find a good soup-to-nuts description of how to build a custom input component, starting with the markup you want, but I figured I could dive in and figure it out.

A Solution

First, the Markup

I have the markup I’m aiming for (above). Here’s the Simple Form field that I think should create it (published_at is the Active Record datetime field to be created/edited):

<%= f.input :published_at, as: :date_time_picker %>

I started with a shell of a custom input component, from the Custom inputs section of the Simple Form documentation. If you put the file in the app/inputs directory (which is Simple Form-specific–you’ll probably need to create it) and restart your server, Simple Form will pick it up:

# app/inputs/date_time_picker_input.rb

class DateTimePickerInput < SimpleForm::Inputs::Base
  def input(wrapper_options)
    merged_input_options = merge_wrapper_options(input_html_options, wrapper_options)

    @builder.text_field(attribute_name, merged_input_options)
  end
end

Looking at the markup I’m shooting for, I’m going to need a text input, and I’m guessing that’s what @builder.text_field is going to create, so let’s just try it out! When I restart my server and render a form using a this custom input component (using exactly the f.input line, above), this is the HTML that’s rendered:

<div class="form-group date_time_picker required content_published_at">
  <label class="control-label date_time_picker required" for="content_published_at">
    Published at
  </label>
  <input class="form-control date_time_picker required"
         type="text"
         value="2018-01-01 23:38:00 UTC"
         name="content[published_at]"
         id="content_published_at" />
</div>

Not a bad start! It looks like Simple Form will generate a label by default, and I don’t want one. Poking around in the documentation and examples, I see that there is a label method that I can define to customize the label tag of this component. What if I define one that returns nothing?

# app/inputs/date_time_picker_input.rb

class DateTimePickerInput < SimpleForm::Inputs::Base
  def input(wrapper_options)
    merged_input_options = merge_wrapper_options(input_html_options, wrapper_options)

    @builder.text_field(attribute_name, merged_input_options)
  end

  def label(_); end
end

And its output:

<div class="form-group date_time_picker required content_published_at">
  <input class="form-control date_time_picker required"
         type="text"
         value="2018-01-01 23:38:00 UTC"
         name="content[published_at]"
         id="content_published_at" />
</div>

Bingo! (The _ parameter is there because the label method requires one parameter, but we aren’t using it and don’t care about it. This is a Ruby convention that’s pretty handy.) Moving on…

I see that the outer div with a class of form-group is already there without me having to do anything. We can ignore the other classes on that outer div, but you can see the name of our component there, the model name/field name (content_published_at) that Rails wants, etc.

Comparing the HTML to the exemplar, the input needs to be wrapped in a div. From some of the examples I saw in the documentation and elsewhere, I can wrap a div around the input using template.content_tag(:div). Let’s give that a shot in the input method:

# app/inputs/date_time_picker_input.rb

def input(wrapper_options)
  merged_input_options = merge_wrapper_options(input_html_options, wrapper_options)

  template.content_tag(:div, class: 'input-group date', id: 'datetimepicker1') do
    @builder.text_field(attribute_name, merged_input_options)
  end
end

Here’s the HTML that’s generated:

<div class="form-group date_time_picker required content_published_at">
  <div class="input-group date" id="datetimepicker1">
    <input class="form-control date_time_picker required"
           type="text"
           value="2018-01-01 23:38:00 UTC"
           name="content[published_at]"
           id="content_published_at" />
  </div>
</div>

Closer!

Thinking about this for a second, the custom component should not be assigning an id to that div–this is a general purpose component. We’ll figure out how to set the id later, if we need it–for now, I’m going to delete the id.

Now I need to create the span that follows the input. And that span needs to contain a nested span inside of it. Hmm–that’s a pattern we’ve seen before. The nested span can be created like this:

template.content_tag(:span, class: 'input-group-addon') do
  template.content_tag(:span, '', class: 'glyphicon glyphicon-calendar')
end

Adding that into the input method, we have:

# app/inputs/date_time_picker_input.rb

def input(wrapper_options)
  merged_input_options = merge_wrapper_options(input_html_options, wrapper_options)

  template.content_tag(:div, class: 'input-group date') do
    @builder.text_field(attribute_name, merged_input_options)
    template.content_tag(:span, class: 'input-group-addon') do
      template.content_tag(:span, '', class: 'glyphicon glyphicon-calendar')
    end
  end
end

When we run this, we get this HTML:

<div class="form-group date_time_picker required content_published_at">
  <div class="input-group date">
    <span class="input-group-addon">
      <span class="glyphicon glyphicon-calendar"></span>
    </span>
  </div>
</div>

Wait! What happened to the input?

Well, the way these tag builder blocks work is that the “return value” of the block is what’s wrapped by the tag. In this case, the return value of the block is whatever the template.content_tag(:span) builder returns, which is the nested spans that we see. The output of the @builder.text_field() call fell on the floor and wasn’t used at all.

The return value of the template.content_tag(:div, class: 'input-group date') builder block needs to be contain both the input and the span, in that order. The input and the span builders just return strings, so we can simply concatenate them! Here’s a simple way to do that:

# app/inputs/date_time_picker_input.rb

def input(wrapper_options)
  merged_input_options = merge_wrapper_options(input_html_options, wrapper_options)

  template.content_tag(:div, class: 'input-group date') do
    input = @builder.text_field(attribute_name, merged_input_options)
    span = template.content_tag(:span, class: 'input-group-addon') do
      template.content_tag(:span, '', class: 'glyphicon glyphicon-calendar')
    end
    "#{input} #{span}".html_safe
  end
end

And the HTML:

<div class="form-group date_time_picker required content_published_at">
  <div class="input-group date">
    <input class="form-control date_time_picker required"
           type="text"
           value="2018-01-01 23:38:00 UTC"
           name="content[published_at]"
           id="content_published_at" />
    <span class="input-group-addon">
      <span class="glyphicon glyphicon-calendar"></span>
    </span>
  </div>
</div>

Awesome!

Because we haven’t hooked up the JavaScript part of this component, it doesn’t do anything, yet. Let’s do that now.

The JavaScript

The bootstrap-datetime-picker home page describes the simple JavaScript that hooks up and gives life to the picker. From near the top of that page, this is all that it takes:

$(function () {
  $('#datetimepicker1').datetimepicker();
});

This JavaScript uses jQuery to call the datetimepicker function (provided by bootstrap-datetime-picker, which is already installed) on the DOM element whose id is datetimepicker1. Whoops–I deleted that!

Our component doesn’t have an id, but does it need one? Let’s say we had a form that had two or more of these pickers in it–we’d want all of them to be turned on. The HTML of our component has plenty of identifying classes attached to various elements of it–we can find all of the picker components in the DOM using those classes.

A slight change to the JavaScript will enable all of the pickers in the DOM:

$(function () {
  $('.date_time_picker > .input-group.date').datetimepicker();
});

This generic JavaScript is written this way so that the code inside of the anonymous function gets called when the page has finished loading. (Exactly how that works is beyond the scope of this post, but a little Googling around should find tons of articles about it.) Rails 5.1 has a different way to run JavaScript when the page is loaded (the details of which are also beyond the scope of this post):

$(document).on("turbolinks:load", function() {
  $('.date_time_picker > .input-group.date').datetimepicker();
});

Here’s the equivalent CoffeeScript (which is the default flavor of JavaScript as of Rails 5.1):

$(document).on "turbolinks:load", ->
  $('.date_time_picker > .input-group.date').datetimepicker()

With that code added to your page’s JavaScript (I’m not going to explain how to do that, but there’s lots of Rails documentation and tutorials online), the picker component works! If you’ve been following along, hopefully you see something like this:

When I started using this picker, it kind of worked, but had some problems.

The first was that when it was used to edit an existing value, the picker would start out blank, instead of populated with the value. When I watched closely, I saw that the right value flickered quickly in the picker, but was erased almost immediately. For some reason, the value does populate the picker initially, but when the datetimepicker() function is called, it gets erased.

My quick fix to that problem is to modify the JavaScript to grab the value of the picker, call the initializer, then restore the value. Seems to work. (There’s probably a better way to do this. Let me know!)

The next problem I saw was that the date that was saved in the Active Record datetime field was a little funky–sometimes it was just a little wrong, sometimes it wasn’t stored at all. Turns out that the format of the date/time string has to match the format that the Rails database expects it. It will try to parse the string using its format, and sometimes it will be just a little wrong, and other times it would be an illegal date, and nothing gets stored!

The way I chose to fix this was to change the format of the string that’s returned by the bootstrap-datetime-picker to match the format that my database (MySQL) expects. (I think the format that MySQL expects–YYYY-MM-DD HH:mm:ss–is fairly common, if not ubiquitous, among databases.)

Here’s the final JavaScript that works well for me:

$(document).on("turbolinks:load", function() {
  const $pickerInput = $('.date_time_picker input.date_time_picker');
  const initialValue = $pickerInput.val();
  $('.date_time_picker > .input-group.date').datetimepicker({ format: 'YYYY-MM-DD HH:mm:ss' });
  return $pickerInput.val(initialValue);
});

Here’s the equivalent CoffeeScript:

$(document).on "turbolinks:load", ->
  $pickerInput = $('.date_time_picker input.date_time_picker')
  initialValue = $pickerInput.val()
  $('.date_time_picker > .input-group.date').datetimepicker({ format: 'YYYY-MM-DD HH:mm:ss' })
  $pickerInput.val(initialValue)

Please leave a note if you have questions, answers, suggestions, or a better solution!

Rendering Dynamic Markdown in Rails

2018-01-27T00:00:00-08:00

I wrote about how I enabled Markdown views in my Rails 5 app previously. I wanted to add the ability to create and edit Markdown-formatted articles, posts, etc. in this app, and it turned out to be pretty easy.

The Markdown that I want to render will be created dynamically by the users of my app and stored in the Rails database, as opposed to being in static .md files somewhere in my views directory. I already had the redcarpet gem installed and configured in my app–I just needed to figure out how to use it to render arbitrary Markdown in a view.

My Markdown handler had changed slightly from its initial incarnation–I had tweaked the Redcarpet settings a little and done a tiny bit of refactoring. Here’s what I started with:

# config/initializers/markdown.rb

require 'redcarpet'

module ActionView
  module Template::Handlers
    class Markdown
      class_attribute :default_format
      self.default_format = Mime[:html]

      class << self
        def call(template)
          compiled_source = erb.call(template)
          "#{name}.render(begin;#{compiled_source};end)"
        end

        def render(template)
          markdown.render(template).html_safe
        end

        private

        def extensions
          @md_options ||= {
            autolink: true,
            fenced_code_blocks: true,
            highlight: true,
            quotes: true,
            strikethrough: true,
            tables: true,
            underline: true,
          }
        end

        def markdown
          @markdown ||= Redcarpet::Markdown.new(renderer, extensions)
        end

        def renderer_options
          @renderer_options ||= {
            filter_html: true,
            hard_wrap: true,
          }
        end

        def renderer
          @renderer ||= HTMLWithPants.new(renderer_options)
        end

        def erb
          @erb ||= ActionView::Template.registered_template_handler(:erb)
        end
      end
    end
  end
end

class HTMLWithPants < Redcarpet::Render::HTML
  include Redcarpet::Render::SmartyPants
end

ActionView::Template.register_template_handler(:md, ActionView::Template::Handlers::Markdown)

Notice that all of the Markdown stuff is private–a Rails template handler just needs to respond to :call–it doesn’t need to expose its implementation details (and shouldn’t). Because of what call returns in this particular handler–a String that gets evaled somewhere in ActionView, and the evaled code calls the render method–the render method needs to be public, too.

What I want to end up with is an application-wide markdown(text) method that accepts Markdown-formatted text and returns HTML that can be displayed by any view. The Redcarpet::Markdown object has a render(text) method that’s just what we need–I just need to make that markdown method in my handler public and use it in the helper! The result:

# config/initializers/markdown.rb

# moved into the public section
def markdown
  @markdown ||= Redcarpet::Markdown.new(renderer, extensions)
end

private

# app/helpers/application_helper.rb

def markdown(text)
  ActionView::Template::Handlers::Markdown.markdown.render(text).html_safe
end

Not the prettiest line of Ruby code (and it violates the Law of Demeter), but it works–and it uses the already-instantiated Redcarpet::Markdown object. I can pretty it up, later.

If you want to render the Markdown from your users differently than the Markdown in your own views, you might want to instantiate (and cache) a Redcarpet::Markdown object with different settings. E.g., you might want to be more strict about what is allowed in the Markdown that your users can enter, compared to what you allow in your own views.

That .html_safe is important–without it, the HTML generated from the Markdown will be “escaped” by Rails, displaying a bunch of ugly, raw HTML in your view.

With this helper, you’re able to put a call to markdown in any views that you like! For example, I have a ContentsController whose show method reads a Content record from the database into a @content variable. The Content model has a body field containing Markdown formatted text, so in the views/contents/show.html.erb file, there’s a markdown(@content.body) call that inserts beautiful HTML from the Markdown text that the user entered!

Please leave a note if you have questions or suggestions.

Adding support for Markdown views to Rails 5

2017-12-19T00:00:00-08:00

I thought that adding Markdown support to a Ruby on Rails 5 project would be a straightforward thing. I’ve read a bunch of blogs over the years that described various ways to use Markdown in Rails apps. There are many Markdown gems–redcarpet, kramdown, rdiscount, to name a few. And there are several ways to use Markdown–from allowing a content management system to author new posts in Markdown to supporting Markdown-formatted views in a Rails app itself. I was interested in the latter, and expected it to be pretty easy.

Wrong.

My first road block was conceptual. I’ve been using Rails for a long time, off and on, but haven’t used it consistently since pre-Rails 3. My recollection, which may be just wrong, is that at some point, Rails’ view rendering allowed chaining of several template renderers–if you named your view something like, posts.html.md.erb, and you had the handlers installed, the view source would first be handled by the ERB handler, then the Markdown handler, then Rails’ standard HTML view processing. Turns out, this is not the way that Rails’ template handlers work (and they may never have worked that way).

As far as I can figure out, ActionView template handlers aren’t chainable– they are intended to convert from one type of template (ERB or Markdown or Haml, say) into the final MIME type that your template is targeting (HTML or JSON or XML, for example), and no more. If you want to create views in Markdown that can have ERB inside of them, then your Markdown handler needs to know how to handle ERB itself.

I searched all over the Web and found several helpful blog posts, but none that did exactly what I wanted (a few came pretty close, though). The Rails Guides don’t discuss how to create your own template handlers, and even the Rails source wasn’t much help to me. I wanted a clean initializer and I wanted to enable ERB in the Markdown templates.

A Solution

I chose to use the redcarpet Markdown gem because it’s fast, feature-rich, and well-maintained. redcarpet needs to be available in all of the Rails app’s environments, so don’t put it into a development-only stanza of your Gemfile! (Been there, done that.)

I like to use TDD (though I’m not super-consistent about it), and it’s easy to create a test for this–just create a view in Markdown and verify that the HTML that’s rendered contains the right tag(s)! I’m using RSpec, and my test looks like this:

# spec/views/welcome/about.html.md_spec.rb

require 'rails_helper'

RSpec.describe 'welcome/about.html.md' do
  it 'renders Markdown' do
    render

    expect(rendered).to match %r{<h3>About My Website</h3>}
  end
end

It will fail, of course, because there is no such view.

Then, create the Markdown view:

<!-- app/views/welcome/about.html.md -->

### About My Website

It's really cool!

The spec will still fail, but in a different way, since now the template is asking for a handler that doesn’t exist.

Finally, the Markdown handler!

# config/initializers/redcarpet.rb

require 'redcarpet'

module ActionView
  module Template::Handlers
    class Markdown
      class_attribute :default_format
      self.default_format = Mime[:html]

      class << self
        def call(template)
          compiled_source = erb.call(template)
          "#{name}.render(begin;#{compiled_source};end)"
        end

        def render(template)
          markdown.render(template).html_safe
        end

        private

        def md_options
          @md_options ||= {
            autolink: true,
            fenced_code_blocks: true,
            strikethrough: true,
            tables: true,
          }
        end

        def markdown
          @markdown ||= Redcarpet::Markdown.new(HTMLWithPants, md_options)
        end

        def erb
          @erb ||= ActionView::Template.registered_template_handler(:erb)
        end
      end
    end
  end
end

class HTMLWithPants < Redcarpet::Render::HTML
  include Redcarpet::Render::SmartyPants
end

ActionView::Template.register_template_handler(:md, ActionView::Template::Handlers::Markdown)

A little explanation, starting from the bottom. ActionView::Template.register_template_handler is the method that registers a new template handler. It expects the filename extension that the handler applies to (:md, for files that end in .md), and an object that implements a call method–in our case, the class that we defined, ActionView::Template::Handlers::Markdown.

The class << self block wraps the class methods, of which only two are public–call and render. (render only needs to be public because it’s going to be called by Rails, which we’ll see in a second.) The magic happens in the call class method–the class itself implements the call method that register_template_handler requires.

The call method needs to return a string containing the code that needs to be executed by Rails to generate HTML (in this template handler, because of the self.default_format = Mime[:html] line). (There isn’t much, if any, documentation on this–I got this from the articles I found online.)

The compiled_source = erb.call(template) line is what handles ERB in the Markdown template. The string it returns is then handled by the Markdown renderer that’s in our code.

"#{name}.render(begin;#{compiled_source};end)" is the return value of the call method–that’s the string of code that Rails will eventually evaluate and run. Because the code will be run in a Rails context, it needs to fully-qualify the method to be run. The name method returns the full name of self, which is the class, so it returns ActionView::Template::Handlers::Markdown. #{name}.render, then, fully-qualifies our render method.

The parameter to render–begin;#{compiled_source};end–is also mysterious. It wraps a Ruby block around the string that the ERB handler returned (that is also Ruby code), which makes it digestible to the Markdown render method.

I’m still a little fuzzy on why the code that invokes the handler needs to be passed around as a Ruby string, but this solution works nicely for me!

Please leave a note if you have questions, suggestions, or a better explanation.

Why can’t Smart Routers be Really Smart?

2017-01-01T00:00:00-08:00

In 2016 the media reported on a number of Distributed Denial of Service (DDoS) attacks that significantly disrupted aspects of the Internet, wreaking havoc that ranged from minor inconveniences for many, to serious financial losses for some. The DDoS attacks were caused by botnets, which are composed of large numbers of unsecured, hacked devices that are connected to the Internet. Attackers take advantage of numerous well-known and lesser-known vulnerabilities in a multitude of connected computers, routers, surveillance cameras, and other devices. When they find a device they can hack into, they take control of the device for their own, often destructive, purposes.

Sales of Internet-connected devices (a.k.a., the Internet of Things or IoT) are exploding, and there are no signs that the trend is slowing. On the contrary, more and more connected devices are being introduced and sold into homes and businesses because they are so useful and/or cool–they are here to stay. Unfortunately, we haven’t yet figured out how to protect these devices from attackers, or how to protect ourselves from hacked devices.

There are thousands of vulnerable products in millions of homes and businesses, and each of them need to be fixed by its manufacturer. That will be a big, distributed effort that will take a long time. Until that happens, how can we prevent attackers from finding and hacking vulnerable Internet-connected devices?

All of the vulnerable devices have one thing in common–they connect to the Internet via a relatively inexpensive, sophisticated, commercial router. Every household and most businesses have one router through which 100% of their Internet traffic flows–you can think of it as an Internet gatekeeper. Sometimes that router is built into the modem that is provided by their Internet Service Provider (ISP), sometimes it’s a separate box connected to the ISP’s modem. Routers are quite complicated–they often have built-in firewalls and VPNs for security, logic to intelligently route traffic to the devices “behind” them and much more.

My question is, why hasn’t one or more of the very sophisticated companies that make and sell routers–Cisco, D-Link, Netgear, ~~Apple~~, Google, etc.–made an inexpensive commercial router that’s really smart? A Really Smart Router (RSR) would have at least these characteristics:

An RSR makes sure that it has a strong password.
An RSR itself is very well protected.
An RSR checks the devices connected to it for known vulnerabilities.
An RSR automatically and securely keeps its software up-to-date.

Each of these features probably needs to be able to be overridden by their owner, for reasons they think are strong enough to reduce the security of their network, but they should be turned on by default.

Strong Password

Almost all routers have a simple/weak password assigned when they are brand new–it’s printed in the router’s manual so that new owners can easily set up their new router. The manual usually advises the new owner IN ALL CAPS to change the password to something stronger, but leaves it at that. People being people, many new router owners never change their router’s password, making it an easy target for attackers.

An RSR would refuse to work until its password was set to something hard for an attacker to figure out. “Hard to figure out” can be satisfied a number of ways. Maybe the password can’t be too short, so that it can’t be guessed quickly. Maybe the password can’t be in a dictionary of common words or have any form of the initial simple password in it. Maybe an RSR doesn’t allow login attempts to happen quickly, so that attackers can’t “brute force” guess thousands or millions of passwords in a very short time.

Well Protected

Once protected by a strong password, I think that most routers are pretty resistant to exploitation by attackers. However, I’m sure that more than a few have known vulnerabilities that haven’t been fixed, or even have “back doors”–ways into the router that bypass the password check that are intentionally built into the router.

An RSR would be a fortress–it would have no back doors, its software would be aggressively tested for vulnerabilities, and any found would be fixed quickly.

Checks Connected Devices

This is a biggie. Many security experts and companies are well aware of almost all of the vulnerabilities that attackers use to create botnets. The experts know what the devices “look like”; they know how to exploit their vulnerabilities.

On some frequency–once a day, once a week, when it detects a new device–a Really Smart Router checks every device connected to it against a comprehensive list of known vulnerabilities. If it finds a vulnerable device, it informs the owner and disconnects the device from the Internet completely. The owner would receive enough information to know what to do next–which device is the problem, what the problem is, and how to fix it. (Some devices are not easily updatable–they may need to be recycled.)

The devil is in the details, but this doesn’t seem like a terribly difficult feature to implement.

An RSR wouldn’t store every known vulnerability inside of itself (even if it had unlimited space, the list of vulnerabilities is always growing)–it would “phone home” to its manufacturer with the “signature” of the devices connected to it, and its manufacturer would send it the list of vulnerabilities to check for on each of them.

I’m frankly surprised that no router manufacturer that I know of does this already.

Automatically Updated

Even a Really Smart Router that did all of these things could not be perfect. New Internet-connected devices are created all the time, maybe with types of vulnerabilities that the RSR doesn’t initially know how to check for. The RSR’s software itself may have vulnerabilities that don’t get discovered until after many of them are installed in customers’ homes and businesses.

Most if not all routers already have a way to update their software. However, not many routers make updating their software easy or convenient, or have an option to automatically update it. People being people, I think RSRs must have an option to automatically check for and install new software without requiring the user to press a button or visit a web page–it needs to be 100% automatic.

The updating process itself needs to be secure, too–it must be impossible for an attacker to pretend that it’s the RSR’s manufacturer, and fool it into downloading and installing the attacker’s software!

Our phones and computers can do this–heck, even smart thermostats update themselves to the latest software automatically! There’s no good reason our routers don’t.

We need Really Smart Routers!

Every year, millions of new connected devices are installed into people’s houses and businesses. DDoS attacks will continue to happen more and more often, causing more and more disruption, unless vulnerable devices are removed from the Internet when they’re discovered.

I think that we need Really Smart Routers, and badly!

What do you think?