ephemeral.cx

Losing Access To The Cascades

Tue, 17 Sep 2024 00:00:00 -0700

Pictured above is the Canyon Creek bridge along the Green Mountain forest road. It and the road beyond were damaged in 2011 and has remained closed ever since. Each year its condition degrades further making any eventual repair/replacement even more expensive. Meanwhile, the bridge closure rendered the remaining 8.5 miles of Green Mountain road inaccessible to vehicle traffic. This transformed the already challenging Three Fingers Lookout trail from 4,200ft and 15mi to a highly committing 5,300ft and 26mi putting it out of reach for all but the most dedicated and skilled hikers. Despite being closed for 13 years, there is little hope for repair and reopening. With each passing year the continued degradation of the road most likely means the loss of access to this area of the Cascades forever.

Even more unfortunately is that this is far from a unique story. The access to trails that thousands of recreational mountain-goers in the Cascades all rely on is slowly eroding due to lack of maintenance on our forest roads. Areas of the Cascades are either becoming considerably more difficult to visit or lost entirely as roads become washed out, closed, and simply never repaired. What’s more, this further exacerbates the overcrowding problems as more people are funneled into the smaller number of trailheads that remain with sufficient access.

Why is this though? It’s not due to any malicious desire to purposefully restrict access but rather one core issue: lack of funding. Specifically, this article aims to explore the history of how our forest road network came to be, the current state of the network, its likely future, and potential mitigations to avoid its continued degradation.

Scope

A short note on the scope of this article and the organization of the USFS: Unlike other government agencies that manage federal lands such as the National Park Service (NPS) and Bureau of Land Management (BLM) which are under the authority of the Department of the Interior, the Forest Service is under the United States Department of Agriculture (USDA). This is a somewhat unusual arrangement and it means that policies, budgets, and priorities may diverge between the USFS and NPS, among others.

As such, this article focuses nearly exclusively on the USFS rather than the NPS. The latter is an entirely different topic with an entirely different set of needs, budgets, and maintenance backlogs which is outside the scope of the analysis performed here.

How did we get here?

To start, it’s safe to say that the forest road network was built on the back of the logging industry of decades past. Absent it we would not have even a fraction of the forest roads that we currently do. Forest roads were rapidly built to support logging operations after WWII through to the 1980s. Afterward, growth virtually stopped in the 1990s corresponding with the decline of the logging industry thus leaving us with essentially the forest road network we know today.

_{Size of forest road network over time ¹}

The end result of this construction being a total network size of approximately 375,000 miles of forest roads nationwide.¹ It’s difficult to comprehend just how massive this network is. To put it in perspective, the forest road network is eight times the size of the entire interstate highway system. In fact, 375,000 miles is greater than the distance to the moon!

_{Forest road network size comparison ²}

It truly is a massive road network. So how, then, was it built?

Prior to WWII, most of the US timber supply was generated from private forests. At that time National Forest land only supplied 2% of US timber. After WWII, this percentage grew as the Forest Service began to raise harvest limits in the 1950s. Harvest rates peaked in the early 1990s before beginning to decline.³ Today, logging levels are a fraction of their peak and back to pre-1950s levels.

_{Logging over time ³}

To understand the implications of this expansion and its subsequent decline we need to understand what timber receipts are and how they fund forest management. In short, timber receipts are money paid by logging companies to the Forest Service for the rights to harvest timber from federal land. These timber receipts in turn go back to the Forest Service as funding for forest management, including, but not limited to, road maintenance. There are a few broad categories this funding goes to: ⁴

The Knutson-Vandenberg Act of 1930 mandates that a certain amount of proceeds from logging be used for reforestation and forest management. The exact amount will vary depending on the needs of a specific area to be cut, but is generally around 25%.
25% goes to the state the logged forest is located in.
10% is contributed to the Roads and Trails Fund for road and trail maintenance.
Roads built for the purposes of logging by the logging company are eligible for credits against the owed timber receipts. In other words, a new road’s construction costs are deducted from the owed receipts stemming from its logging operations.
Any remaining funds are sent to the general fund of the US Treasury.

The end result being that during the decades when the logging activity in National Forests was high there was a large amount of funding for forest road maintenance being generated through timber receipts and incentives for logging companies to build new roads by deducting their construction costs from the owed receipts. Accordingly, as logging in National Forests rapidly declined during the 1990s so did the revenue collected from timber receipts. Consider exactly how much funding was lost between the 1990s and today:

Year	Gross Timber Receipts	Roads Funding
1980	$1.9B	$195M
1985	$558M	$56M
1990	$1.6B	$161M
1995	$369M	$37M
2000	$187M	$19M
2005	$248M	$25M
2010	$136M	$14M
2015	$200M	$20M
2019	$186M	$19M

_{Roads funding over time ⁵}

We can see from those numbers that, on average, 10% of timber receipts went back to road maintenance and repair. Today, the overall funding from this source has declined by upwards of 90% from its peak in the 1980s and 1990s.

Despite this funding drop for maintenance, as of the 1990s the amount of recreation use in the national forests has climbed by 1,400% from 1950s levels, and is surely considerably higher today.

_{Forest recreation use over time ¹}

Budgets & Appropriations

Such a large decrease of road funding would make the future of these roads seem bleak. There’s more to this story than the decline of timber receipts, however. For a deeper understanding, we must delve into the exciting world of federal budgets!

As all government agencies have, the Forest Service has an annual budget appropriated by Congress that comes from tax revenue. The part of the USFS budget we’re interested in here is the Capital Improvement and Maintenance fund, also known as the CIM budget. Between 2010 and 2014, CIM funding dropped by 37% before stabilizing after 2015.⁶

_{Appropriated CIM budget 2010 - 2018 ⁶}

Unfortunately, the Forest Service explicitly states that this level of CIM funding only allows them to repair issues that pose an immediate health and safety risk.⁶ Everything else is effectively unmaintained.

Region 6 & Mt. Baker-Snoqualmie National Forest

It’s here that I want to switch from looking at budgets at the national level to the regional level to better understand what’s going on at a more local and relatable level. Specifically, let’s look at the PNW forest region and the Mt. Baker-Snoqualmie National Forest (MBS) as this is the region I live in and the forest that I and many of my fellow Seattle-area residents spend most of our time in.

One more housekeeping note on the organization of the Forest Service: The USFS is divided into nine regions, R1-R10 (region 7 does not exist anymore). Each region is then divided further into National Forests and each Forest is divided into Ranger Districts. Each of these units have their own budgets.

_{Forest Service regional organization}

With that said, let’s take a look at the exactly how much funding is needed to maintain the roads in Region 6 (the PNW). The table below lists the deferred and annual maintenance costs for each forest in Region 6 as of 2015.

The difference between deferred and annual maintenance being:

Deferred Maintenance: Maintenance that was not performed when it should have been or was put off or delayed for a future period. When allowed to accumulate without limits or consideration of useful life, deferred maintenance leads to deterioration of performance, increased costs to repair, and decrease in asset value.
Annual Maintenance: Work performed to maintain serviceability or repair failures during the year in which they occur. This includes preventive and/or cyclic maintenance performed in the year in which it is scheduled to occur.⁷

Below is the deferred maintenance for 2015 & 2023 and annual maintenance need for 2015 (the most recent year for which I have data) for each forest in Region 6: ⁷ ⁸

National Forest	Road Miles	Annual Maintenance Need (2015)	Deferred Maintenance (2015)	Deferred Maintenance (2023)
Columbia River Gorge	99	$121,557	$1,454,584	$454,671
Colville	4,309	$4,306,765	$37,336,065	$44,657,021
Deschutes	8,109	$7,526,877	$80,566,681	$29,767,043
Fremont-Winema	12,548	$13,642,507	$133,971,908	$68,925,716
Gifford Pinchot	4,103	$5,312,486	$53,330,891	$27,182,755
Malheur	9,628	$6,153,833	$56,025,932	$24,603,985
Mount Hood	2,881	$4,896,610	$51,813,990	$25,269,040
Mt. Baker-Snoqualmie	2,453	$9,660,568	$81,915,920	$62,880,382
Ochoco	3,253	$3,313,734	$33,260,537	$14,980,293
Okanogan-Wenatchee	8,163	$17,050,400	$158,111,026	$84,572,497
Olympic	2,026	$4,467,995	$42,680,614	$27,172,794
Rogue River-Siskiyou	5,288	$11,581,995	$111,614,953	$72,534,097
Siuslaw	2,128	$2,777,636	$26,115,387	$16,091,493
Umatilla	4,624	$6,647,168	$65,211,612	$32,083,051
Umpqua	4,776	$7,148,103	$73,669,140	$36,795,738
Wallowa-Whitman	9,150	$6,808,709	$64,279,905	$30,869,516
Willamette	6,542	$8,838,067	$90,942,456	$47,717,136
Total	90,078	$120,255,010	$1,162,301,600	$646,557,228

_{The caveat here is that the source of this data notes these 2015 numbers are to return the road network to a “like new” condition. For practical purposes we don’t necessarily need “like new” conditions everywhere so an actual number may be somewhat smaller but regardless this makes for a decent ballpark upper-bound figure.}

Wait a second though, the numbers above show that as of 2015 the deferred maintenance for the region was estimated at just over $1.1B⁷ compared to $646M in 2023. Progress on the backlog must be being made then, right? Unfortunately not. This is not because the backlog is being caught up with but rather because roads are simply being closed or downgraded in quality thus getting the deferred maintenance they incurred off the books. Consider the following proposed distribution of road maintenance levels from 2015: ⁷

Due to the lack of funding the percentage of closed roads was proposed to be doubled. The roads accessible to low-clearance, passenger cars (PC) reduced by a third and the roads accessible to high-clearance vehicles (HC) halved.

This is also a good time to mention that even if you are lucky enough to have a high-clearance vehicle and are happy with certain roads being harder to pass as a way of controlling crowds, this lack of maintenance funding still affects your access as formerly high-clearance-only roads degrade to the point of being impassable to any vehicle and closed entirely. It is also more expensive to rehabilitate a road that was unmaintained or under-maintained. Preventative maintenance is almost always cheaper in the long run and the obstacles to re-opening a closed road are extremely high.

As an aside, the Forest Service categorizes roads into five maintenance levels with levels 5, 4, and 3 being suitable for low-clearance, passenger vehicles, level 2 being suitable for high-clearance vehicles, and level 1 being closed to traffic.

_{Road maintenance levels examples, levels 5 - 1 left to right ⁹}

A short term solution to addressing budget shortfalls, therefore, is to reduce the maintenance level of roads or close them outright. This reduces the deferred maintenance backlog at the expense of losing access entirely to areas served by now closed roads or reducing access to only high-clearance vehicles. Given the lack of funding this is what the Forest Service has been forced to do in recent years. Because of that, the annual maintenance need is correspondingly likely somewhat lower now from 2015 but not by an appreciable amount, especially when factoring in inflation.

In the case of Mt. Baker-Snoqualmie (MBS) specifically, ignoring the backlog of deferred maintenance, just shy of $10M/year is needed to maintain its roads.⁷ How much funding is the forest getting then? The average annual road maintenance budget between 2008 and 2012 was $810k.⁷ By 2013 that number was down to $400k.¹⁰

A clear trend starts to emerge here:

In 1998, the Forest Service estimated they had funding to maintain 40% of the network.¹
With the collapse of timber receipts after the 1990s, at 2008-2012 funding levels that was down to 25%.¹⁰
As noted above, by 2013 further reduced CIM funding took it down to 7% if efforts were focused on solely on a small subset of roads to maintain to standard.⁷

Yikes.

Wildfires

Despite all of this, it’s notable that the Forest Service’s overall national budget actually increased from $5.1B to $8.2B between 2011 and 2020.¹¹ Even adjusted for inflation, this represents a 36% increase in funding. So what’s going on here? Why has the budget increased by so much in real dollars but the maintenance budgets keep getting slashed?

This additional funding has primarily gone to wildfire fire fighting. And even with that increase fire fighting needs have diverted funding from other programs. The wildfire account in the budgets is named “Wildland Fire Management” or WFM. In the chart below, it’s evident how much the WFM funding has increased from 2011 to 2020 while the maintenance budget (CIM) shrunk from 2011 levels (although it did noticeably grow starting in 2018 which we’ll get back to):

_{Forest Service Discretionary Appropriations by Account, FY2011-FY2020 ¹¹}

Or rather in table format: ¹¹

Year	CIM Budget	WFM Budget
2011	$459.6M	$2.0B
2012	$382.1M	$2.0B
2013	$346.5M	$2.5B
2014	$333.0M	$3.0B
2015	$343.4M	$2.6B
2016	$348.2M	$3.9B
2017	$348.0M	$3.1B
2018	$525.6M	$3.4B
2019	$467.0M	$3.7B
2020	$466.8M	$4.3B

While the CIM budget dipped between 2011 and 2017 before returning to 2011 levels in 2018, the WFM budget was more than doubled by 2020. Of course, this is understandable given the crisis levels of wildfires we have been dealing with over the past decade. The issue is that these emergencies have risen to the level of creating another budgetary issue: fire borrowing.

The Congressional report Forest Service Appropriations: Ten-Year Data and Trends (FY2011-FY2020) states the following regarding fire borrowing:

Overall appropriations to the FS for wildfire-related activities have increased considerably since the 1990s. A significant portion of that increase is related to rising suppression costs, even during years of relatively mild wildfire activity, although the costs vary annually and are difficult to predict.

Due to the emergency nature of fire control activities, appropriations laws provide the FS authority to transfer money out of other discretionary accounts if suppression funds become depleted; this is often referred to as fire borrowing. When such transfers have occurred, Congress typically has enacted supplemental appropriations to repay the transferred funds and/or to replenish the agency’s wildfire accounts, though sometimes these funds have been provided in subsequent fiscal years.¹¹

In recent years efforts have been made to reduce the impact of fire borrowing on other areas of the budget, but the practice remains. Even when diverted funds are subsequently replaced this borrowing still creates unpredictability which has the potential to derail any project planning to address deferred maintenance. In the meantime, the agency has directly noted the impact of fire borrowing on the CIM budget up to this point:

Complicating the increased constraints on CIM funding, large amounts of the Agency’s funding must be allocated to fight large wildfires. Wildfire suppression operations have had a substantial impact on the Agency’s ability to financially plan, design, and implement responses to its growing portfolio of unmaintained infrastructure assets. Rising costs and fire borrowing have diverted much-needed funds away from CIM efforts, thereby increasing the delays and costs of deferred maintenance projects. As a solution, Congress enacted a wildfire cap adjustment that will virtually eliminate the need for fire borrowing starting in FY2020.⁶

_{The last statement about the wildfire cap introduced in 2020 is notable and hopefully this will therefore be less of an issue in the future, but it is still important to note fire borrowing to understand how the maintenance backlog became so large.}

It’s not just budgetary effects either. In 1995, there were 18,000 staff working in the Forest Service plus another 5,700 fire personnel. By 2015 general staff decreased to 11,000 while fire personnel increased to 12,000.¹² Even with funding provided for maintenance in a given year, this leaves far fewer employees to make use of that funding as all the money in the world doesn’t mean much if there’s an insufficient staff to put it to use.

Overall, in 1995 16% of the Forest Service budget was dedicated to wildfires. By 2015 it was 52% and by 2025 it’s projected to be upwards of 67%.¹² Without large amounts of additional funding it is virtually guaranteed that the Forest Service’s budget will continue to be siphoned away by firefighting needs.

Recreation Passes

Now at this point you may be thinking “hang on, I pay for a Northwest Forest Pass/America The Beautiful Pass each year to use all these trails. Isn’t that paying for these trails and roads?” And that is an excellent question. These do provide a non-trivial amount of revenue, but not as much as you may think and it doesn’t necessarily go towards what you may think it does.

_{…you did buy a pass before parking at the trailhead, right?}

The Forest Service publishes data specifically on how much revenue the Recreation Fee Program brings in and where it goes each year. For 2022 the breakdown for Region 6 and the Mt. Baker-Snoqualmie Forest (MBS) was: ¹³ ¹⁴

Revenue	Region 6	MBS Forest
Recreation Fees	$10.1M	$1.1M
Interagency Passes (America the Beautiful Pass)	$959k	$234k
Special Uses	$1.7M	$221k
Total	$12.8M	$1.6M

Expenditures	Region 6	MBS Forest
Repair and Maintenance	$5M	$857k
Visitor Services	$2.5M	$139k
Law Enforcement	$109k	$50k
Habitat Restoration	$7k	$0
Fee Agreements	$12k	$0
Collections/Overhead	$917k	$71k
Total	$8.7M	$1.1M

It’s hardly covering all of the needed costs, but it’s certainly not nothing! However, for MBS specifically it was shown earlier that the annual maintenance need is in the neighborhood of $10M whereas the annual appropriations budget was around $400k after 2013. So while that $857k directed towards repair & maintenance comprises a huge chunk of the overall maintenance revenue, the forest still has only around 13% of what it needs.

Additionally, it is worth noting that the Forest Service primarily uses these funds for trail maintenance rather than roads.¹⁵ Which is certainly important, but since this article is focused on forest roads it’s important to note that the revenue collected from the recreation fee programs is almost entirely dedicated to trail maintenance rather than road maintenance. And the trails are of no use if there’s no way to get to them so both trail and road maintenance are critical.

Other Funds

Despite all of this, there is a big piece of the puzzle that has been left out until this point. Annual appropriations don’t tell the whole story about funding; there are additional supplemental government permanent or temporary programs that will provide non-trivial amounts of funding for individual projects. Any analysis of forest road funding would be massively incomplete without considering them so let’s take a look at the main ones.

_{A note: This is not meant to be an exhaustive list of programs that provide forest road funding. Tracking down every single source of funding for the Forest Service is a task well outside the wheelhouse of my little blog. The goal here was to touch on the main/current ones to get a general idea of funding levels.}

Roads and Trails Fund

Let’s start with one of the oldest funds: The Roads and Trails Fund was established in 1913 and allows for 10% of revenues generated from uses of National Forests to be used for construction and maintenance of roads and trails in the forests.¹¹ For example, the aforementioned timber receipts.

That sounds great, except that since 1982 appropriations laws have instead directed all of these revenues to the general fund of the US Treasury instead.⁴ Alright, what else we got?

Legacy Roads and Trails

The appropriately named Legacy Roads and Trails program was first created in 2008 with the goal of repairing, well, roads & trails, decommissioning roads, and removing fish passage barriers. Each year this fund must be reauthorized with varying levels of funding. For example, in 2013 it was funded with $39M¹⁶ yet in 2018 it was funded with $40M.⁵ As in, inflation has taken a toll on funding levels here.

This fund is a fraction of what is needed, however. For instance, in 2011 this funding maintained and improved just over 1,500 miles of road across the entire network.¹⁶ Keep in mind that the forest road network is around 375,000 miles in total. So that covers 0.4% of the network. Granted, this was only a single year and, sure, it’s better than nothing, but funding to maintain under 1% of the network is hardly going to move the needle on keeping up with needed maintenance let alone catching up on deferred maintenance.

Bipartisan Infrastructure Law

In 2012, the Federal Lands Transportation Program (FLTP) was first established by the Moving Ahead for Progress in the 21st Century Act. It provided $300M/yr in funding for transportation-related facilities on federal lands administered by the National Park Service (NPS), Fish and Wildlife Service, and Forest Service. It was reauthorized in 2015 by the Fixing America’s Surface Transportation Act and most recently again in 2021 by the Infrastructure Investment and Jobs Act, colloquially known as the Bipartisan Infrastructure Law of 2021. By 2026 FLTP will provide $456M in funding.¹⁷

That’s actually a fairly significant amount! Except… the overwhelming majority of this funding goes to the NPS. Of that $456M to be allocated in 2026, $360M will go to the NPS, $36M to the Fish and Wildlife Service, and $28M to the Forest Service.¹⁷

Of course, the NPS badly needs funding as well, but the point is that the FLTP also will not appreciably reduce the Forest Service’s deferred maintenance backlog. As noted at the start of this article, this is not an article about the NPS so the impact of this funding on their backlogs is out of scope here.

Aside from re-authorizing FLTP, the Bipartisan Infrastructure Law does not provide much in the way of funding for road maintenance. It does provide $5.5B for the Forest Service to “restore ecosystems and reduce wildfire risk to communities” as well as “important tools to support agency efforts to establish resilient landscapes for future climate conditions.”¹⁸

Inflation Reduction Act

The 2022 Inflation Reduction Act provided an additional one-time $5B appropriation for additional wildfire resilience measures.¹⁸ As far as I understand, this funding is not to help with road maintenance but was worth mentioning here due to its recency and high-profile nature.

Great American Outdoors Act

Finally we arrive at the 2020 Great American Outdoors Act (GAOA). This act was the most consequential law in recent years for improving the maintenance funding situation and general access to outdoor recreation. In the most broad strokes the GAOA consists of two main components:

It established permanent funding for the Land and Water Conservation Fund (LWCF). This fund existed since 1964, but lacked permanent yearly funding. GAOA established a permanent $900M/yr funded by offshore oil and gas drilling royalties (as was the case since the fund’s inception).¹⁹
It created the Legacy Restoration Fund with $9.5B of funding over five years. This fund is targeted at addressing maintenance specifically.¹⁹

For example, since at the time of this writing it is 2024 we are already most of the way through the Legacy Restoration Fund’s life. We can therefore see some of the projects that GAOA has accomplished already. Below is a list of GAOA-funded projects for the Mt. Baker-Snoqualmie forest specifically:

Forest	Project	GAOA Non-transportation funding	GAOA transportation funding
MBS	Mountain Loop Highway Corridor Enhancement	$225k
MBS	Pacific Crest Trail Access Roads, Bridges, and Trails Deferred Maintenance	$1.3M	$200k
MBS	Bridge Repairs and Preservation		$600k
MBS	Heather Meadows Trails and Recreation Site Deferred Maintenance and Dam Rehab	$60k
MBS	Mountain Loop Highway Road, Trail, and Bridges Deferred Maintenance	$450k	$1M

_{See Appendix D in ¹⁸ for a full list of GAOA-funded projects in Region 6}

This all sounds great, but as usual the devil is in the details so what’s the asterisk on this?

For the LWCF, its funding is split between federal use to acquire land and grants to states for projects of their choice.²⁰ Both important uses, but not ones relevant to addressing the deferred maintenance backlog on federal lands.

As a small digression, a local example of LWCF funding used for state-specific projects, Seattle readers of this article have the LWCF to thank for part of the development of Gas Works park in years past:

LWCF funding for Gas Works Park is an investment in Seattle’s quality of life that has paid dividends. […] Without LWCF funding, the City would have been unable to develop the 2-acre northwest portion of the park, which had been inaccessible and vacant for many years.²¹

For the Legacy Restoration Fund, the issue is that $6.65B of the allocated $9.5B of this goes to the NPS rather than the USFS.²² The Forest Service can get at most $285M/yr or just shy of $1.5B over the lifetime of the five years of the fund. Moreover, the GAOA stipulates that 65% of each agency’s funding must be used for non-transportation projects²³ (hence the distinction in the table above).

Remember that the deferred road maintenance backlog in Region 6 alone was $646M as of 2023. For simplicity assume this $1.5B is spread equally over all nine USFS regions and then take 35% of that for the allowed quota of transportation projects. That’s ~8% of the total deferred road maintenance for Region 6. Which is certainly something, but still leaves 92% of the backlog unaccounted for and is only funded for five years, which we are already one year from the end of as of this writing.

Again though, this is not to diminish the needs of the NPS over those of the USFS. So far $173M of the NPS’s share of GAOA funding has gone to multiple much needed projects in Washington.²²

Budget Summary

Alright, so after analyzing all of those sources of funding, what conclusions can we draw about the state of funding for our forest roads? Overall, there are a few high level takeaways up to this point:

The traditional source of forest road maintenance funding, timber receipts, dramatically dried up with the collapse of the logging industry in the 1990s.
Since then, appropriations to maintain the existing roads have been further reduced leaving the Forest Service able to maintain only a small percentage of the roads.
Meanwhile the deferred maintenance backlog has only grown forcing the Forest Service to permanently close roads or downgrade their maintenance level.
In recent years, funding has been diverted to more pressing wildfire fighting needs.
Recent laws aimed at addressing these maintenance backlogs of federal recreational facilities have been directed primarily at the NPS rather than the USFS.

Therefore, the unfortunate reality is that the current level of funding for our forest roads remains woefully inadequate. With this level of funding we can expect to see more roads falling further into disrepair, becoming inaccessible to most vehicles, and ultimately being closed & lost forever regardless of what vehicle you may drive.

Let’s now turn to how we can avoid this future.

Potential Mitigations

In an ideal world the annual appropriations from Congress would cover the needed annual maintenance budget plus some extra to work through the deferred maintenance backlog, but this is unrealistic to expect. The federal budget has enough strain on it already and the USFS is hardly the only agency feeling this pinch. As valuable as I honestly believe outdoor recreational access to be for the public, there are higher priorities than maintaining some roads in a forest. So let’s assume then that the USFS annual appropriations are what they are and won’t be dramatically increasing any time soon. What can be done to prevent losing these roads forever in that case?

There’s no silver bullet solution here unfortunately. In reality, a combination of approaches are needed to get funding to where it is needed. What’s more, getting funding to 100% of what’s needed is likely even more unrealistic. But that’s not an excuse to not attempt to at least improve the situation and why this section is titled “mitigations” rather then “solutions.”

Maybe instead of being able to maintain a paltry ~5% of the roads, we could maintain 50% of them. With maintenance efforts focused on the most popular roads for recreational access we would at least not be slowly losing access to trailheads entirely.

1. Extend the GAOA Legacy Restoration Fund

An easy win would be to extend the Great American Outdoors Act’s Legacy Restoration Fund (LRF) for another five years. Above it was mentioned that the GAOA established the LRF with $9.5B of funding over five years. This was in 2020 so as of this writing there is one more year left in this program. Extending it for another five year term would provide a significant amount of additional funding for working on the deferred maintenance backlog. Perhaps make it a ten year term or maybe even give it permanent funding? Given it passed the House with a 2/3 majority and the Senate with a 3/4 majority,²⁴ it should be “easy” (by Washington standards) to get an extension passed.

The LRF’s funding is currently heavily weighted towards the NPS. Ideally, the funding would be more evenly distributed between the NPS and the USFS as well so that the USFS may have adequate funding to work through its maintenance backlog too.

2. Use the Land and Water Conservation Fund for maintenance

In addition to establishing the Legacy Restoration Fund, the GAOA also provided a permanent source of funding for the Land and Water Conservation Fund (LWCF) to the tune of $900M/yr. The LWCF is primarily used for land acquisition, however. Some of its funding could instead be directed to deferred maintenance each year instead.

Prior to the GAOA, while the fund was authorized to collect and dispense up to $900M/yr, many years it was distributing much less.²⁵ Rarely did appropriations exceed even $500M. With the GAOA it is now fully funded each year leaving a significant amount of money that was previously not being fully utilized by the LWCF.

_{LWCF Annual Discretionary Appropriations (in millions, not inflation adjusted)²⁵}

That’s not to say it should be entirely used for maintenance over land acquisition, but it could be argued that maintenance needs of already-owned federal lands trumps new acquisitions moreso now than when the fund was established in 1964. Needs change over time and the laws written to meet those needs should adapt as well.

Ideally the LWCF’s funding would be pegged to inflation too to ensure it does not become further eroded in the future now that it is fully funded.

3. Increase enforcement of recreation passes

As discussed above, revenue from recreation passes (Northwest Forest Pass, America the Beautiful Pass) for Mt. Baker-Snoqualmie National Forest in 2022 was $1.6M.¹⁴ Now this is anecdotal, but it seems that at most trailheads only around half of vehicles have a pass displayed on them. At the more popular trails this number can be even lower, or some people mistakenly believe that Washington’s Discover Pass is valid on federal land. Moreover, in all my years hiking the Cascades I have seen the Forest Service enforcing recreation passes at a trailhead a single time.

Given that recreation passes contributed $1.6M in funding already, increased enforcement of these passes represents an opportunity to increase funding significantly simply by enforcing laws already on the books. Yes, enforcement itself costs money and getting to 100% is unrealistic, but as it stands nearly no enforcement at all leads many to forgo passes as it’s known the chances of getting caught without one are so low. A small amount of increased enforcement (and fines for those without passes) has the ability to collect a non-trivial amount of additional funding that could be used for maintenance of these facilities.

4. Raise the cost of recreation passes

This option is a difficult one to propose. First and foremost, the objective is not to price anyone out, which I’ll address in just a moment. Here’s the thing though: the cost of a Northwest Forest Pass (NWFP) and interagency (America The Beautiful) pass are simply too low.

Consider the NWFP: When it was introduced in 1997 a yearly pass cost $25. Today it is $30/yr. If the cost of the pass was simply kept consistent with inflation it would be $49/yr. Remember that the recreation fees brought in $10M in revenue for Region 6 in 2022 ¹³ and that the annual road maintenance in 2015 was $120M/yr. Adjusting the cost for inflation would yield an additional ~$6.5M in revenue. Not that this alone is going to solve the problem, but it’s a non-trivial amount of funding and only keeps the cost consistent with what it was historically.

Now consider the timeframe the NWFP was first introduced. This was in the mid-1990s, a time when the logging industry had only just started to retreat taking much of the timber receipt funding with it. The maintenance backlog was less and the funding situation for annual maintenance was much less dire. $25/yr may have been reasonable back then. The fiscal environment is entirely different now. As much as we would all love for the government to fund maintenance or for some resource extraction industry to pay for it, the reality of the situation is that roads cost money and that money has to come from somewhere. If other sources of funding are not forthcoming it would fall on the users of these roads to pay for them lest we allow them to go unmaintained and accept they will disappear. Assuming that’s not the conclusion, it would be reasonable to accept increased fees to pay for these roads. Thus, realistically the cost of recreational passes should be considerably higher to 1. adjust them for inflation, and 2. pay for at least some of the annual maintenance needs to help supplement other revenue sources. Exactly what that number should be is outside of scope here, but the point is that the current fees are simply much too low.

This is absolutely not to say that we should purposefully be gatekeeping low income people out of the forests. There are few options here: Have a cheaper rate for those below a certain income threshold, make a certain number of passes available for checkout at local libraries, or offer free passes to volunteers that do trail work or trash pickups (as the NPS already does).

5. Strengthen the Roads and Trails Fund Act

The Roads and Trails Fund, mentioned earlier in this article, was established in 1913 to provide funding for construction and maintenance of roads and trails in national forests. Specifically 10% of all revenue generated from forest products would be directed back to the forest’s roads and trails.²⁶ While this funding is permanent, unfortunately, the phrasing of this act is fairly weak and does not strictly require that 10% of funds be directed back to the forests but rather “may whenever practical” be directed back to the forests. As such, since 1982, this funding was sent to the general fund of the Treasury to offset annual appropriations for road/trail maintenance.⁴

During my research for this article I found one reference to the 2020 President’s Budget proposing that the revenue collected by the Roads and Trails Fund once again be used for its intended purpose.⁶ It is not clear if this was enacted, however, as I could not locate any further information on if this proposal was enacted. Regardless, a better long term solution would be to reform the 1913 act to mandate that revenue collected is used for road and trail maintenance rather than only “whenever practical.”

6. Make use of the Highway Trust Fund

In 1956 the Federal Aid Highway Act created the Highway Trust Fund to finance interstate highway (and other road) construction and maintenance. It is funded by a 18.5 cent per gallon tax on gasoline. Since forest roads are federal roads on federal land and the Highway Trust Fund is presently used not only for interstate highways but also surface roads and the gas used to travel on these roads is subject to the federal gas tax it would not be unreasonable to provide funding to maintain them from the Highway Trust Fund too. Considering the federal gas tax revenue was over $50B in 2023²⁷ and the road maintenance needs for Forest Service Region 6 were $120M in 2015⁷, this is a fairly small amount.

There is a caveat to this one though. The federal gas tax is already insufficient to fund the Highway Trust Fund. Between inflation, increased fuel efficiency of cars, and electric vehicles, the revenue going towards the Highway Trust Fund has been declining. The federal gas tax itself has not been raised since 1993 despite inflation halving the value it collects in real dollars since then. In fact, since 2008 over $275B in general tax revenue has been transferred to to the Highway Trust Fund to keep it solvent.²⁸ As gasoline use declines a new solution will need to be found for road funding. The federal gas tax also needs to be raised to at least be consistent with inflation. Anything that makes gas more expensive is a political lightning rod, but if you don’t pay taxes for roads at the gas pump, general tax funds are used instead. Either way you are being taxed to pay for roads; nothing is free and roads are no exception. The pending insolvency of highway funding is a separate topic entirely, however.

7. Allow more concessionaires

For reference, a concessionaire is a private business that has a contract with the USFS or NPS to provide services or operate facilities in a forest or park. For instance, many of the sightseeing tours, retail/lodging operations, food services, etc. in our National Parks are operated by private concessionaires rather than the NPS directly.

How does this help with road maintenance then? Commercial operations have direct lines of revenue and are only able to operate with acceptable levels of infrastructure to their area of operation. This means it is essential to maintain the areas they operate in. For example, consider Cascade Powder Guides. They are a cat skiing company that operates near Stevens Pass on USFS land. Access is through forest road 6710 and their revenue is what pays for winter plowing of (a portion of) that road. No matter what happens, as long as they are in operation, that road will remain well maintained.

More concessionaires would mean more businesses with strong interests in maintaining access to their respective areas of operations. Perhaps small lodges could be built at certain trailheads. This would be useful for road maintenance of course, but also with providing more options for people to enjoy the mountains.

To be clear, there is a balance here. This is not a proposal to start building massive luxury hotels along every forest road. Rather it is a proposal to allow some well-balanced development in areas where it is already legally feasible (as in, not wilderness areas) as a source of revenue for forest management and recreation. I understand that this is a fine line to walk, but keep in mind that if this line was not walked in the past we would not have our National Parks as they exist today as they are a tradeoff between development/recreational access and preservation.

Take the Sperry Chalet in Glacier National Park for example. The chalet is a backcountry hotel (meaning it is hike-in only) located within the park originally built in 1914 by the Great Northern Railway. With only 17 rooms, the chalet is a small operation that does not drive disproportionate amounts of traffic to its area of the park. It also serves as a example for how modern construction can remain true to historic construction methods as the hotel was faithfully rebuilt following a 2017 fire that destroyed the building.

There are many options here and what makes sense in one area may not make sense in another. But consider that in addition to providing revenue for maintaining a certain area, concessionaires provide access to the mountains for people that otherwise may not be able to access areas to due physical limitations. In the example of the Sperry Chalet, access via horse is possible for those physically unable to make the 3,000+ vertical feet hike to the chalet. It’s important to keep in mind that access to our mountains is for more than only those capable of hiking thousands of vertical feet with a 40lbs backpacking pack on.

8. Increase sustainable logging

Taken at face value suggesting to increase logging would seem like an absurd proposal. However, there is a compelling case to make for increased logging.

As shown at the start of this article, the amount of logging done today is a fraction of what it historically was. That’s not to say it would be wise to return to peak levels of logging we once saw, but rather that the current levels of logging are at historic lows and it is possible to increase logging while still being sustainable. The main benefits to this are:

Timber receipts were directly responsible for the initial build-out of the forest road network. Increased logging would once again fund road maintenance.
Surprisingly enough, carbon sequestration rates are higher on private, managed forests than in public forests despite these private forests having higher levels of logging.²⁹ Increased sustainable logging on public lands is not a forgone environmental catastrophe as many would make it out to be.
Modern sustainable logging techniques, not clear cutting entire areas, thins forests reducing their susceptibility to massively damaging wildfires.
Many areas that were harvested in decades past are ready for second or even third harvests. To be abundantly clear, this is not a proposal to start clearing never logged, old growth forests again.
- After all, many of these roads were originally built for logging access. By definition, they lead to areas that were already logged. So logging them again is not cutting pristine forest land. And the areas beyond the roads are generally highly protected wilderness areas where logging is completely prohibited; that does not change.

In fact, this is already happening. In 2018, 3.1 billion board feet of timber was harvested. This is the highest level since 1999,⁶ but still far shy of the peak levels of ~12B board feet. We are learning how to log more sustainably than in decades past and that is an unequivocally good thing.

Mt. Pilchuck case study

To illustrate this proposal with a concrete example, let’s take a look at the Mt. Pilchuck road as an example of how logging helps improve recreation access. The road to the Mt. Pilchuck trailhead was originally built as a, you guessed it, logging road in the 1950s. Its terminus then served as the parking lot and base area for the Pilchuck ski area until its closure in 1978 (in fact, I’ve previously written about the feasibility of re-establishing winter access to Pilchuck noting that the main roadblock was the poor state of the road). Since then, the Mt. Pilchuck trail and the lower elevation Heather Lake trail have become two of the most popular hikes along the Mountain Loop Highway corridor. The problem was that prior to 2024, the Mt. Pilchuck road had become the poster child for poorly maintained forest roads in that corridor. It was so bad that news articles were written about it.

But then in 2023 the road was closed for the entire summer while it was fully repaired with the Heather Lake trailhead parking tripled in size. When it reopened for the 2024 season the road had been transformed into one of the best forest roads your esteemed author has ever driven on.

What happened to accomplish this? Well, the road was not repaired because of annual maintenance funding or any other federal funding source. Rather, it was part of the South Fork Stillaguamish Vegetation Management Project. This project encompassed an area much larger than just along the Pilchuck road.

As can be seen from the map above, this logging project is directly responsible for road/bridge repairs and trailhead upgrades all along the Granite Falls side of Mountain Loop Highway.

It was not without its share of controversy and legal battles, however. In 2020 the North Cascades Conservation Council filed a lawsuit against the USFS arguing, among other items, that the proposed logging was too heavy, the NEPA (environmental) review was inadequate, and that there would be a net positive amount of roads constructed going against the 1994 Northwest Forest Plan³⁰ (the 1994 plan directs projects to have an overall net reduction or at least neutral impact on the size of the forest road network).

There’s a lot to unpack with this case that would be off-topic for this article. The short of it is that the USFS noted the need for thinning of these areas due to maintain forest health by:

Thin[ning] previously harvested stands that are currently 20 to 80 year second growth stands of age within the SF of the Stillaguamish River drainage. The goals of the stand treatments are to promote old forest characteristics with large diameter trees, stands with multiple layers of canopy, and the retention of down wood and snag components. The proposed thinning also has the goals of enhancing species diversity, maintaining a high rate of growth on dominant trees, developing desired vegetation stocking and diversity in Riparian Reserves, and promoting desired growth and stand conditions across the landscape for resiliency to climate change.³¹

As for the inadequate NEPA review, all of the documentation for the NEPA review is publicly available. You can be the judge of if it was inadequate or not. I’ve previously written about the weaponization of NEPA reviews by special interest groups that simply want to throw up roadblocks to any and all projects (NIMBYs in other words). This is unfortunately another classic example of that having served no purpose other than to waste time and money that could have been used for additional forest management projects.

Here in 2024 the project is thankfully well underway despite a 2022 appeal. There is already a freshly repaired Pilchuck road to show for it and an expanded Heather Lake trailhead. And that is only along one of the roads involved in the project. This area now has an ongoing source of maintenance funding, better road drainage protecting downstream water resources, improved recreational access/facilities, lower wildfire risk, a healthier forest that more closely resembles the old growth forest before logging began in the 1940s, and the economic benefits of the jobs provided by the mills in Darrington. It’s a win-win all around.

Imagine applying that same playbook to other areas in need of road, trail, and forest maintenance.

Conclusions

The harsh reality here is that this topic does not have a happy ending. The current maintenance funding of the forest road network is woefully and borderline hopelessly inadequate. The high level summary of this article being:

The forest road network was built during the heyday of the logging industry and funded primarily through timber receipts.
With logging now back to pre-1950s levels, the traditional source of funding for road maintenance has largely dried up.
The current funding is only able to maintain 7% of the roads.
This has left us with a massive deferred maintenance backlog and an annual maintenance budget that is only sufficient to maintain a fraction of the road network.
Due to this budget shortfall the forest service has been forced to downgrade roads’ maintenance levels or close them entirely.
Despite all of that, there are many possible mitigations to generate additional funding to maintain at least a larger percentage of our roads if not all of them.

So, if you’re wondering why that one road to the hike you want to do is seemingly taking forever to be repaired, this is why.

To bring this full circle, this article started with an example of the Canyon Creek bridge washout (pictured right). Despite the bridge itself being in good shape, 13 years of neglect has made it susceptible to collapse due to erosion under its north abutment. The current path we are on ends with either its removal or collapse and the permanent loss of the road beyond it and potentially the trail entirely if it is deemed too dangerous to ford the river without a bridge. I imagine a world where access to the road beyond the bridge is restored via a combination of, say, sustainable logging, perhaps a small commercial lodge further up the road, or any number of other creative means of funding. The possibilities are there.

At these present levels of funding we can expect access to trails that thousands of hikers, backpackers, mountaineers, climbers, ski tourers, and more regularly utilize to slowly become less accessible before becoming lost entirely. In many ways, our mountains were more expansive and accessible in decades past. Today we have fewer options than predecessors did and when outdoor recreation was less popular. With an ever growing Washington population, this loss of access will have the side effect of forcing more recreationalists into a smaller collection of trailheads further exacerbating the overcrowding issues our mountains already experience. At a time when we should be looking at increased dispersion to alleviate this concern the exact opposite is happening.

With all of that in mind, none of this should be seen as an insurmountable problem, however. It is one that can solved with a sufficient amount of political will for making the necessary reforms to provide funding. There are many avenues for providing sufficient or at least improved maintenance funding. We need only advocate for a figurative and literal road forward.

Sources

Hey Google, what happened to all the fun?

Thu, 11 Jul 2024 00:00:00 -0700

This is the story of how Google killed a 14 year old Android app overnight.

2008 was a time when the web had mostly become ubiquitous but still before most people carried it all with them in their pocket on a smartphone. For me, a high school student at the time without a smartphone, my programming classes were the only times during the school day where I could access the internet in a school computer lab. These short periods during the day were often filled writing programs of various computer science fundamentals for my classwork but, of course, there was also a healthy amount of screwing around on the heavily filtered internet we were allowed access to.

It was in one of these computer labs that a fellow student directed me to a website with quite literal domain, isittuesday.com. It was exactly what it sounded like, a large “Yes!” or “No.” displayed on the page if it was Tuesday and, well, that’s it. It was the sort of random website that you’d snicker at, send to your friends on AOL Instant Messenger for the next person to snicker at, and move on to the next thing that caught your brief interest.

For those of us that grew up during this time, the web was still a fairly decentralized place. Terms like “Web 2.0” and “the blogosphere” abounded. The social media giants we know today were becoming established, but the web did not revolve around them just yet. But I am hardly the first person to express nostalgia for this era. So what?

_{Credit: xkcd}

Well, silly websites like Is It Tuesday? exemplified people having fun with the web. There was no monetization, no data collection, no purpose to it other than to put smiles on faces. Instead of the infinitely scrolling, ad-infested feeds of proprietary smartphone apps like we have today, browser extensions like StumbleUpon and websites like The Useless Web were excellent ways to entertain one’s self or waste time.

Fast forward two years to 2010. Smartphones were becoming the norm. I myself received my first Android phone that year, a Motorola Droid, and became fascinated at what could now be done on my phone. The Google Play Store, or Android Market as it was known at the time, was starting to fill up with all sorts of apps. From the useful like SSH clients to the useless like Staples Easy Button simulators.

_{The only screenshot I have from my first Android phone and all the weird apps I had installed.}

Naturally I wanted to put my own app up on the Android Market. It helped that most of my limited programming background as a then high school student was in Java so it made it relatively easy to start writing Android apps. After some brainstorming about a simple first app to write, that Is It Tuesday? website from years ago came to mind. I went about implementing it, which took about an hour since, as you can guess, is pretty darn simple. In fact, spoiler alert, here’s basically the entire app:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
public class main extends Activity {
  @Override
  public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.main);
    setLabel();
  }

  @Override
  protected void onStart() {
    super.onStart();
    setLabel();
  }

  @Override
  protected void onResume() {
    super.onResume();
    setLabel();
  }

  private void setLabel() {
    Calendar cal = Calendar.getInstance();
    TextView textView = (TextView) findViewById(R.id.label);

    if(cal.get(Calendar.DAY_OF_WEEK) == Calendar.TUESDAY) {
      textView.setText(R.string.yes);
    } else {
      textView.setText(R.string.no);
    }
  }
}

_{Fun fact: This app did have a bug initially. At first I did not implement onResume(). So if the app was opened on a Monday, sent to the background, but not fully closed, and then opened again the next day, it would still display “No” rather than “Yes” on a Tuesday. How awful!}

This was also the time that publishing an app to the Android Market was quite easy. Google was eager to differentiate themselves from how Apple was handling their App Store. It was free to publish an app and there was no explicit approval process gatekeeping the way. You made a developer account, filled out a basic form, uploaded an APK, and you were live. Or at least that’s how I remember it—it was simple enough for a high school student with a rudimentary programming background to figure out.

And thus, on February 9th 2010 my first app went live. For someone still in high school this was actually an exciting moment in my life. It was the first time that software I wrote was being distributed to users, to people other than me and my teachers at school, to run on hardware that was not mine. And wouldn’t you know it: 14 years later, it’s still there! You can even still install it for the time being if you’re so inclined.

I told some friends about it, but otherwise there was no marketing or advertising. Still, I could see from the downloads metrics that people were installing it for some reason. Then the reviews started coming in.

Of course, these were people having just as much fun with the reviews as they did with the app similar to the reviews for the Three Wolf Moon shirt on Amazon around the same time.

So yes, the whole app was entirely pointless, it was silly, it was useless. But it made people smile and the reviews some of those people left made other people smile. In short, it was fun.

For years I would occasionally check in on it to read the reviews myself and get a chuckle out of the things that people would write. But beyond that, it was just a silly app I published years prior that obviously did not require much upkeep. At one point I went about seven years without updating it:

As everything in the world does though, time moved on and the Android ecosystem evolved. The Android Market became the Play Store and the requirements for publishing apps increased as Google sought to wrangle back control of the Android platform from the open nature it was founded on. I had noticed that the app was so old that it was not showing up in the search results for newer Android devices due to being built with such an early API version.

In recent years I would occasionally update it just to bump the target SDK version. Google started requiring developers to fill out disclosure forms for things like health data collection, user age verifications, etc. I even had to create a privacy policy website just to state that the app did not collect any information. The hoops to jump through became numerous enough that I thought of no longer updating it and letting it fade into the ether. But each time I ended up rolling my eyes at whatever new requirement Google added and complying with it to keep the app alive. After all, it was fun and at this point I was motivated to keep the streak alive since it had been on the Play Store since nearly its inception.

Then on July 8th 2024 I received an email stating that I needed to update the target SDK version before the end of August to keep the app published. While the code of the app hadn’t changed since 2010, I had been updating it to work with the current Android build tools so bumping the the target SDK version and compiling a new APK binary was no big deal, I thought.

Unfortunately, now, unlike in 2010, all Android app updates now have to go through an approval process. In the past this hadn’t been an issue. Google would do whatever they did for this approval process and Is It Tuesday? would be updated within a day. However, this time I woke up the next day to an email stating that my app update had been rejected because it was in violation of the “Minimum Functionality policy.”

In other words, despite being on the Android store for 14 years and going through this approval process multiple times in recent years, Google suddenly decided that the app doesn’t have enough functionality to be allowed on their store anymore. I submitted an appeal explaining that the seeming lack of functionality was the whole point, that it was a “just for fun” app, and that adding whatever extra functionality to make Google’s reviewers happy would defeat the purpose of the app. I wasn’t expecting anything from this since these “appeal” processes are essentially just lip service. Unexpectedly, later that day I received an email saying that my appeal was denied.

I should mention here that I find it ironic Google decided my app lacked any useful functionality and blocked it from being updated while simultaneously leaving up the existing version of the app with that same supposed lack of functionality. Huh?!

So… I guess that’s it. The almightly Google has decided that after 14 years this app no longer deserves to be on their platform. Since I cannot update the target SDK version, as of the August 31st deadline it will disappear from the Play Store. There’s nothing I can do having exhausted the “appeal” process and being unable to comply with whatever the “minimum functionality policy” entails as it would ruin the absurd humor of the app.

Maybe you agree with Google here. After all, why would you want an app store to be full of junk/spam apps that don’t do anything? Putting aside that the app stores are already full of junk apps, I do have sympathy for that argument since no one likes sifting through an ocean of spam. But it’s here that we get to the point of this blog post: This wasn’t an app that did nothing and had no value. Yes, it was simple and it was stupid, I’ll be the first person to admit that. But it made people laugh and that alone creates value, lack of complex functionality be damned. This is the concept that Google is missing. That these “useless” apps aren’t useless. They can be fun and at least some people enjoy them. The reviews alone clearly prove that.

However, this gets to the even larger issue here. It’s easy to understand why so many are attracted to the siren call of these platforms from the large tech companies: they’re built out, they scale, and they have a huge audience you want/need access to. But when you build your business or your livelihood on top of one of these platforms you are ceding your control and your freedom to them. You are essentially signing up to be an employee of sorts with no protections. Whether it be an Uber/Lyft driver, a YouTube/TikTok creator, an app developer, etc.; the company decides to remove you from the platform for any trivial reason, or no reason at all, and you’re done with no recourse, with no one to even make your case to other than maybe some textarea in an “appeal” form submitting to some anonymous person that you know won’t change anything. This isn’t unlike a traditional job, of course, where you can be terminated or laid off for no reason out of the blue, but unlike a traditional job there’s little to no transferability of your work/skills to another platform/employer, if there even is one, and no legal protections since you were never a true employee or even an independent contractor in the first place. You were just another cog that the machine chewed up, spat out, and happily kept chugging along without.

Now this is just a little trivial app so perhaps I’m being melodramatic here. After all, I’m not exactly losing sleep over any of this beyond writing what is essentially a post-mortem for an app that I probably feel a little too connected to given its death represents an era of the internet that I have a lot of nostalgia for. This experience does, however, serve to strengthen my long standing resolve that the free web (independent web servers outside the control of these large tech companies) remains the most open, free (as in freedom), and versatile distribution channel for software and information. Call me overly principled, naive, or idealistic but software freedom is the hill that I will die on.

What’s left for Is It Tuesday? then? I may try to get it published on Fdroid. Otherwise, while the original isittuesday.com website I remember appears to have gone down in 2009, it looks like someone has taken up the mantle with http://isittuesday.co.uk. The joke will live on even without Google’s approval.

The real kicker here is that a similar app is present on the Apple App Store. I guess that means Apple is the more fun one now, Google.

Debugging Ruby, The Hard Way

Mon, 13 Nov 2023 00:00:00 -0800

Normally when you encounter a bug with Ruby, or any other interpreted language for that matter, using the language’s provided debugging tools are all you need to diagnose the problem and find a solution. Indeed that works 99% of the time. But what about when it doesn’t? What about when your program is so hosed that the typical debugging tooling doesn’t yield any fruitful information?

This was the situation I found myself in recently while debugging a low-level bug with Ruby. I didn’t know it when I started, but the problem lie down in glibc and all the Ruby-land debugging tools in the world would not help me. So what’s one to do? Well, if you’re running the C implementation of Ruby, MRI, then it’s GDB to the rescue. However, figuring out how to access the data needed through GDB presents a host of new challenges. Armed with the proper knowledge though and it becomes entirely feasible to debug a Ruby program through GDB which is what this post aims to explore.

But why would you want to do this?

That’s a good question. In all honesty and as far as I know, there’s very few true use cases for doing this outside of development on Ruby itself, academic curiosity, and the poor souls facing a low level bug that’s seemingly impossible to debug otherwise.

In my situation I was working with a Ruby process that would deadlock while exiting in a glibc function in rare cases. I did not have the ability to debug the Ruby process directly as it was completely unresponsive due to control being outside of Ruby when it deadlocked. The only option I had was to attach GDB to the running process in order to get visibility into the process. As I’ll get into later, this provided enough information to put the pieces of the puzzle together and solve my issue at hand. Hopefully this is not what brought you here, but if so, knowing how to debug Ruby via GDB can be a powerful tool in your toolbelt for cracking difficult low-level bugs.

GDB Basics

As someone that primarily works with Ruby and other high level languages, prior to my aforementioned boggle it had been more than a handful of years since I needed to debug anything with GDB. You may be in a similar boat so let’s start with covering enough of the basics to follow along with the rest of this post. If you’re adept with GDB already you can likely skip to the next section.

GDB is, of course, a debugger, and an extremely powerful one at that. We need only know the absolute basics here though. Let’s say we have the following C program:

1
2
3
4
5
6
7
8
9
10
11
12
#include <stdio.h>

const char* GREETING = "Hello, world!";
void print_greeting(const char* greeting);

int main(void) {
  print_greeting(GREETING);
}

void print_greeting(const char* greeting) {
  puts(greeting);
}

When compiling, ensure that -ggdb is specified to create debugging symbols. It’s still possible to debug a binary without these, but it is more difficult and requires referencing the source code more. Make your life easy and add them.

1
$ gcc -ggdb hello_gdb.c

We can then debug the print_greeting function with GDB by setting a breakpoint and executing the program as follows:

1
2
3
4
5
6
7
8
9
10
11
$ gdb a.out
Reading symbols from a.out...

(gdb) break print_greeting
Breakpoint 1 at 0x115f: file hello_gdb.c, line 11.

(gdb) run
Starting program: a.out 

Breakpoint 1, print_greeting (greeting=0x555555556004 "Hello, world!") at hello_gdb.c:11
11        puts(greeting);

Printing a backtrace is accomplished with where:

1
2
3
(gdb) where
#0  print_greeting (greeting=0x555555556004 "Hello, world!") at hello_gdb.c:11
#1  0x000055555555514c in main () at hello_gdb.c:7

The other notable action is assigning variables and executing functions. For example, let’s say we wanted to print out the value of a global variable or call another function through GDB:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ gdb a.out
Reading symbols from a.out...

(gdb) break main
Breakpoint 1 at 0x113d: file hello_gdb.c, line 7.

(gdb) run
Starting program: a.out 

Breakpoint 1, main () at hello_gdb.c:7
7         print_greeting(GREETING);

(gdb) call GREETING
$1 = 0x2004 "Hello, world!"

# `p` also works for printing values
(gdb) p GREETING
$2 = 0x2004 "Hello, world!"

# Manually call the `print_greeting` function
(gdb) call print_greeting(GREETING)
Hello, world!

# Call `print_greeting` with a different argument
(gdb) call print_greeting("Hello, GDB!")
Hello, GDB!

The output above demonstrates how call can be used to print the value of variables (or more accurately, memory locations such as the global variable GREETING). It can also, ahem, call functions directly and we can pass whatever arguments we want to them such as a new string rather than the GREETING string as defined in the source code of the program. Later on we’ll use this functionality extensively to peer inside of a running Ruby process.

That should about do it for a GDB crash course. Let’s get into the meat of exploring Ruby through GDB now.

Getting a Ruby Backtrace

First things first, how can we use GDB on a Ruby process to see what our program is doing? In other words, how do we get a Ruby backtrace out of GDB?

To start, say we have the following simple Ruby program that sleeps for a while:

1
2
3
4
5
6
7
8
#!/usr/bin/env ruby

def foo
  puts 'Sleeping...'
  sleep 1000
end

foo

Using GDB it’s trivial to get a native backtrace as such:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ gdb --args ruby sleep.rb
(gdb) run
Starting program: ruby sleep.rb

(gdb) where
#0  0x00007ffff763446c in ppoll () from /usr/lib/libc.so.6
#1  0x00007ffff7af471a in rb_sigwait_sleep (th=th@entry=0x55555555d040, sigwait_fd=sigwait_fd@entry=3, rel=rel@entry=0x7fffffffb9a0)
#2  0x00007ffff7af58b8 in native_sleep (th=<optimized out>, rel=0x7fffffffb9a0)
#3  0x00007ffff7af8c39 in sleep_hrtime (fl=2, rel=<optimized out>, th=0x55555555d040) at thread.c:1325
#4  rb_thread_wait_for (time=...) at thread.c:1408
#5  0x00007ffff7a4995b in rb_f_sleep (argc=1, argv=0x7ffff7430088, _=<optimized out>) at process.c:5219
#6  0x00007ffff7b31ae7 in vm_call_cfunc_with_frame (ec=0x55555555e1c0, reg_cfp=0x7ffff752ff10, calling=<optimized out>)
#7  0x00007ffff7b419c1 in vm_sendish (method_explorer=<optimized out>, block_handler=<optimized out>, cd=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>)
#8  vm_exec_core (ec=0x7fffffffb8e8, ec@entry=0x55555555e1c0, initial=1, initial@entry=0)
#9  0x00007ffff7b474e3 in rb_vm_exec (ec=0x55555555e1c0, jit_enable_p=jit_enable_p@entry=true) at vm.c:2374
#10 0x00007ffff7b488c8 in rb_iseq_eval_main (iseq=<optimized out>) at vm.c:2633
#11 0x00007ffff7957e75 in rb_ec_exec_node (ec=ec@entry=0x55555555e1c0, n=n@entry=0x7ffff7fbda48) at eval.c:289
#12 0x00007ffff795e4db in ruby_run_node (n=0x7ffff7fbda48) at eval.c:330
#13 0x0000555555555102 in rb_main (argv=0x7fffffffbf18, argc=2) at ./main.c:38
#14 main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57

But that’s not especially helpful in telling us where our Ruby program is hanging. How do we use GDB to get a Ruby backtrace? Ruby actually makes this quite simple, just call rb_backtrace.

1
2
3
4
(gdb) call rb_backtrace()
        from sleep.rb:8:in `<main>'
        from sleep.rb:5:in `foo'
        from sleep.rb:5:in `sleep'

That was easy! This will print the current backtrace to stderr. Done, right? Well, yeah, possibly actually. If you’re just looking to run a Ruby program and get a backtrace at some point during its execution then this should do you fine for the most part. But if that were the case you could also likely set a breakpoint with a Ruby debugger and get the same info with less hassle. What if you have an already running process that’s hung somewhere that you want to get a backtrace for it? That’s a more interesting situation.

In the above example, the rb_backtrace method will print to stderr of the Ruby process, not the GDB process. To demonstrate this, let’s run our Ruby process in one shell and then attach GDB to it from another:

Shell 1:

1
2
$ ./sleep.rb 
Sleeping...

Shell 2:

1
2
3
$ sudo gdb -p `pgrep -n ruby`
(gdb) call rb_backtrace()
(gdb)

Back in shell 1:

1
2
3
4
5
$ ./sleep.rb 
Sleeping...
        from ./sleep.rb:8:in `<main>'
        from ./sleep.rb:5:in `foo'
        from ./sleep.rb:5:in `sleep'

What happened here? stderr is pointed at shell 1, so when we call rb_backtrace() from shell 2, the backtrace is printed out in shell 1. That’s fine for a simple example like this, but if you have an already running Ruby process you need to debug then it’s probably running as a service so stderr isn’t pointed at your terminal. Maybe it and stdout are going to a log somewhere, but let’s assume we don’t have access to them at all. How do we get our backtrace?

To solve this we need to do some more work with GDB before calling for the backtrace. Using GDB we can re-open stderr for the Ruby process to a temporary file, get our backtrace, and then reset them.

Shell 1:

1
2
$ ./sleep.rb 
Sleeping...

Shell 2:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ sudo gdb -p `pgrep -n ruby`
(gdb) set $old_stderr = (int) dup(2)
(gdb) set $fd = (int) creat("/tmp/backtrace.txt", 0644)
(gdb) call (int) dup2($fd, 2)

(gdb) call rb_backtrace()

(gdb) call (int) dup2($old_stderr, 2)
(gdb) call (void) close($old_stderr)
(gdb) call (void) close($fd)

(gdb) quit

$ cat /tmp/backtrace.txt
        from ./sleep.rb:8:in `<main>'
        from ./sleep.rb:5:in `foo'
        from ./sleep.rb:5:in `sleep'

What we did above was set the stderr file descriptor for the Ruby process to a file located under /tmp, called rb_backtrace() to write the backtrace to that file, and then reset stderr to its original file descriptor. This way we can see what our Ruby process is currently doing even if we don’t have access to its stderr stream and without needing to stop the already-running process.

What about other threads?

This is good and all for these simple examples, but most Ruby programs of any complexity will have multiple threads. And if you’re resorting to debugging your program like this then it’s most likely debugging a difficult to reproduce deadlock situation. rb_backtrace() will only print the backtrace of the current thread so how do we get the backtrace of all threads?

To explore this, let’s use a new example Ruby program that starts two threads which deadlock:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/usr/bin/env ruby

mutex1 = Mutex.new
mutex2 = Mutex.new

thread1 = Thread.new do
  mutex1.lock
  sleep(1) until mutex2.locked?
  mutex2.lock
end

thread2 = Thread.new do
  mutex2.lock
  sleep(1) until mutex1.locked?
  mutex1.lock
end

# Start a third thread just so Ruby doesn't recognize the process as deadlocked and kill it before we can debug it
Thread.new {sleep(1000)}

thread1.join
thread2.join

From within GDB again, we can now print all of the running threads with info threads, switch to another thread, and print its backtrace as such:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ ruby deadlock.rb

# In another shell:
$ sudo gdb -p `pgrep -n ruby`

(gdb) info threads
  Id   Target Id                                             Frame 
* 1    Thread 0x7ff76b4637c0 (LWP 1993106) "ruby"            0x00007ff76ab5c4c6 in ppoll () from /usr/lib/libc.so.6
  2    Thread 0x7ff765f3f6c0 (LWP 1993116) "deadlocked.rb:6" 0x00007ff76aae24ae in ?? () from /usr/lib/libc.so.6
  3    Thread 0x7ff765d3e6c0 (LWP 1993117) "deadlocked.rb:*" 0x00007ff76aae24ae in ?? () from /usr/lib/libc.so.6
  4    Thread 0x7ff765b3d6c0 (LWP 1993118) "deadlocked.rb:*" 0x00007ff76aae24ae in ?? () from /usr/lib/libc.so.6

(gdb) thread 2
[Switching to thread 2 (Thread 0x7ff765f3f6c0 (LWP 1993116))]
#0  0x00007ff76aae24ae in ?? () from /usr/lib/libc.so.6

(gdb) call rb_backtrace()

# In the first shell:
        from ./deadlocked.rb:9:in `block in <main>'
        from ./deadlocked.rb:9:in `lock'

Now we can see how the second thread is waiting to acquire the lock for mutex2 at line 9, which, of course, it will never get but it does pinpoint where the program is becoming stuck.

This is a bit tedious to do for every thread though, especially if you have more than just two. It’s fairly easy to automate this process to get a backtrace for every thread, however. Let’s modify the stderr redirection from above and write it to a GDB script named backtrace.gdb this time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
set $old_stdout = (int) dup(1)
set $fd = (int) creat("/tmp/backtrace.txt", 0644)
call (int) dup2($fd, 1)

set $thread_list = rb_thread_list()
set $num_threads = rb_num2long(rb_ary_length(rb_thread_list()))
set $i = 0

while $i < $num_threads
  call rb_p(rb_thread_backtrace_m(0, 0, rb_ary_entry($thread_list, $i++)))
end

call (int) dup2($old_stdout, 1)
call (void) close($old_stdout)
call (void) close($fd)

The changes to this script below are two fold:

Redirect stdout rather than stderr since we’ll be calling rb_p (Ruby’s print function) which prints to stdout now.
Get every thread with rb_thread_list and for each thread call rb_thread_backtrace_m to print a backtrace for that thread in particular.

Running it:

1
2
3
4
5
6
7
8
$ sudo gdb -p `pgrep -n ruby` -x backtrace.gdb
(gdb) quit

$ cat /tmp/backtrace.txt
["./deadlocked.rb:21:in `join'", "./deadlocked.rb:21:in `<main>'"]
["./deadlocked.rb:9:in `lock'", "./deadlocked.rb:9:in `block in <main>'"]
["./deadlocked.rb:15:in `lock'", "./deadlocked.rb:15:in `block in <main>'"]
["./deadlocked.rb:19:in `sleep'", "./deadlocked.rb:19:in `block in <main>'"]

Great! We can see above how each thread has a backtrace printed out in an array format and exactly where each of them are hung.

Into the VM

Everything above should be sufficient for all practical purposes. But if you’ve come this far you may be interested in exploring the Ruby VM a bit more while we’re in here. For instance, those backtraces, how does Ruby track what it’s currently executing in order to generate a backtrace and how can we peek into those data structures of a running process ourselves?

It’s worth nothing everything beyond this point is not necessarily practical per se; it’s primarily academic but still quite interesting to those curious about Ruby internals. Everything below targets Ruby 3.2.2 as well (the current version of MRI at the time of this writing).

There are various blog posts from prior to circa 2017 that demonstrate how to get reach into Ruby’s internal data structures by use of a global variable named ruby_current_thread. However, evidently this was removed in Ruby 2.5.0. Instead, from Ruby 2.5.0 to at least 3.2.2 (at the time of this writing), ruby_current_ec is the new variable to use for this purpose.

What is ec though? ec stands for execution_context and holds the data related to whatever Ruby is executing at a given point in time. This includes the call stack, control frame pointer, thread pointer, fiber pointer, and more. The rb_execution_context_struct definition has the full list of fields, but the short of it is that everything we’re interested in is located in this data structure either directly or through a pointer to another data structure.

Let’s take a look at what this looks like for the sleep.rb script from above (protip: use set print pretty on in GDB for easier to read outputs here):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(gdb) call ruby_current_ec
$1 = (struct rb_execution_context_struct *) 0x5571d51461c0

# `ruby_current_ec` is a pointer so in order to print the data it points to we need to tell GDB to dereference the pointer with the dereference operator
(gdb) call *ruby_current_ec
$2 = {
  vm_stack = 0x7f2316230010,
  vm_stack_size = 131072,
  cfp = 0x7f231632fed0,
  tag = 0x7ffd5b32c250,
  interrupt_flag = 0,
  interrupt_mask = 0,
  fiber_ptr = 0x5571d5146170,
  thread_ptr = 0x5571d5145040,
  local_storage = 0x0,
  local_storage_recursive_hash = 139788682901840,
  local_storage_recursive_hash_for_trace = 4,
  storage = 4,
  root_lep = 0x0,
  root_svar = 0,
  ensure_list = 0x0,
  trace_arg = 0x0,
  errinfo = 4,
  passed_block_handler = 0,
  raised_flag = 0 '\000',
  method_missing_reason = MISSING_NOENTRY,
  private_const_reference = 0,
  machine = {
    stack_start = 0x7ffd5b32d000,
    stack_end = 0x7ffd5b32bf50,
    stack_maxsize = 8372224,
    regs = {
      {
        __jmpbuf = {93947394543840, -6100235198789993538, 0, 93947394543680, 93947394547664, 139788684347000, -122019239599424578, -3910558443652162},
        __mask_was_saved = 0,
        __saved_mask = {
          __val = {0 <repeats 16 times>}
        }
      }
    }
  }
}

It’s best to compare the struct definition against what GDB prints out to get a better idea of what these fields are exactly. But even still, the above doesn’t tell us a whole lot about what our program is doing. In order to determine that we’ll need to explore some of the pointers to other data structures.

In particular, the control frame pointer, cfp, gives us access to what Ruby is executing in this thread. A control frame in Ruby is the data structure that the VM uses to track both your program’s stack and it’s own internal stack in YARV. YARV itself is an entirely different topic that’s outside the scope of this post, but in short it’s the internal bytecode interpreter for Ruby. It has its own internal stack for managing the execution of bytecode instructions. A control frame holds pointers to both of these stacks.

Inside of an execution context the control frame pointer points to the current control frame which is represented as a call stack we’re all familiar with as such:

So, if we have a pointer to the current control frame we can get access to our program’s stack which in turn tells us what is being executed. With that in mind, let’s take a look at the CFP for our deadlocked process:

1
2
3
4
5
6
7
8
9
10
11
(gdb) call *(ruby_current_ec->cfp)
$3 = {
  pc = 0x0,
  sp = 0x7f23162300a8,
  iseq = 0x0,
  self = 139788671962360,
  ep = 0x7f23162300a0,
  block_code = 0x0,
  __bp__ = 0x7f23162300a8,
  jit_return = 0x0
}

Hmm, it doesn’t look quite right that the pc and iseq values are NULL pointers. pc being program counter and iseq instruction sequence. The struct definition is handy to reference for understanding these again. What’s going on here?

If we look into the ep (environment pointer) value we can see that its flags denote this control frame as a VM_FRAME_FLAG_CFRAME rather than a “Ruby frame.” This would track with how our program is calling sleep meaning control passes from Ruby code into an internal C function that in turn makes a syscall. Ruby has a function named VM_FRAME_RUBYFRAME_P which checks the flags on the environment pointer value to determine what type of frame it is. Using the following in GDB we can verify that it is considered a CFRAME by means of being a non-zero value from the bitwise AND operation which is the same operation that VM_FRAME_RUBYFRAME_P ends up performing through another function, VM_FRAME_CFRAME_P:

1
2
(gdb) call ruby_current_ec->cfp->ep[0] & VM_FRAME_FLAG_CFRAME
$4 = 128

That’s fine then, but how do we get a Ruby frame in that case? Because the stack is a contiguous block of memory, it’s easy enough to move up one control frame to inspect the previous frame by simply adding one byte to the pointer value as such:

1
2
3
4
5
6
7
8
9
10
11
(gdb) call ruby_current_ec->cfp + 1
$5 = {
  pc = 0x5571d53fa308,
  sp = 0x7f2316230080,
  iseq = 0x7f2316dfc880,
  self = 139788671962360,
  ep = 0x7f2316230078,
  block_code = 0x0,
  __bp__ = 0x7f2316230080,
  jit_return = 0x0
}

We can also double check this frame’s type is Ruby code by getting the frame type from its environment pointer’s flags and comparing it to the frame type constants (values are printed in hex to more easily compare the constant values):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Frame type of the current control frame
(gdb) p/x ruby_current_ec->cfp->ep[0] & VM_FRAME_MAGIC_MASK
$6 = 0x55550001

# Frame type of the previous control frame
(gdb) p/x (ruby_current_ec->cfp + 1)->ep[0] & VM_FRAME_MAGIC_MASK
$7 = 0x11110001

# The current control frame matches the constant for a C function
(gdb) call (ruby_current_ec->cfp->ep[0] & VM_FRAME_MAGIC_MASK) == VM_FRAME_MAGIC_CFUNC
$8 = 1

# The previous control frame does /not/ match the constant for a C function
(gdb) call ((ruby_current_ec->cfp + 1)->ep[0] & VM_FRAME_MAGIC_MASK) == VM_FRAME_MAGIC_CFUNC
$9 = 0

# The previous control frame /does/ match the constant for a normal Ruby method
(gdb) call (ruby_current_ec->cfp->ep[0] & VM_FRAME_MAGIC_MASK) == VM_FRAME_MAGIC_METHOD
$10 = 0

The exact constant values and bit mapping for the flags are available in the vm_frame_env_flags enum.

But anyway, that control frame looks better now. It has a valid a program counter and instruction sequence pointer. Still though, how do we get that elusive line number to where our Ruby program stopped executing?

Tracking Down a Line Number

Inside the control frame, the instruction sequence value (iseq) is of particular interest since that holds the source location of the currently executing code. We can use this to track down what piece of Ruby code that control frame was executing. Indeed, if we print out the value of the iseq then we see an interesting struct named code_location:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
(gdb) call *((ruby_current_ec->cfp + 1)->iseq->body)
$11 = {
  type = ISEQ_TYPE_METHOD,
  iseq_size = 6,
  iseq_encoded = 0x55a4e9d86c80,
  [...snipped...]
  location = {
    pathobj = 140272768374920,
    base_label = 140272768375800,
    label = 140272768375800,
    first_lineno = 3,
    node_id = 6,
    code_location = {
      beg_pos = {
        lineno = 3,
        column = 0
      },
      end_pos = {
        lineno = 5,
        column = 3
      }
    }
  },
  [...snipped...]

So that’s it then?! Well, no. The beginning position is listed as line 3 and the end position listed as line 5. That corresponds with the foo method in our sleep.rb program, but not the specific line the sleep call is on. This struct instead represents the scope or block that Ruby is executing in. In order to get the line number of the current line of code we have to do a bit more work and call the rb_vm_get_sourceline function:

1
2
(gdb) call rb_vm_get_sourceline(ruby_current_ec->cfp + 1)
$12 = 5

Hey, that was easy! Line 5 indeed corresponds to the sleep call so we know we’re at the right place now. But how did rb_vm_get_sourceline get that from just the control frame pointer? Many of the helper functions that rb_vm_get_sourceline utilizes are inlined so we can’t easily call them directly in GDB. What it boils down to though is that Ruby will do some pointer math with the current program counter pointer and the pointer to the encoded instruction sequence to calculate a position offset and then read into a bit-vector that represents the specific instruction we’re interested in. The relevant definition of this bit-vector can be found in the iseq.c file, but a fair warning that it is heavy on the bitwise operations. The short of it is that this function will pull out the relevant iseq_insn_info_entry struct and then from there it can easily read the line_no field.

Getting a File Name

We have a line number now, but that’s only half of the puzzle. In order to make sense of this we also need to know what file that line number refers to. In this case it’s simple because we only have one file, but let’s pretend that we have a bunch of source files and don’t know which one to look at. How do we get the file name?

If we jump back to the iseq body print output from above, there’s a pathobj field inside the location struct. From the definition of this struct we know that this is either a string or an array with two elements, the relative and absolute path, to the source code file.

1
2
3
4
5
6
7
8
typedef struct rb_iseq_location_struct {
    VALUE pathobj;      /* String (path) or Array [path, realpath]. Frozen. */
    VALUE base_label;   /* String */
    VALUE label;        /* String */
    int first_lineno;
    int node_id;
    rb_code_location_t code_location;
} rb_iseq_location_t;

Before we get into what’s inside that pointer we need to take one more detour first and discuss how Ruby represents strings and arrays internally. The strings and arrays in this case are not your run of the mill C strings & arrays but rather Ruby’s wrappers around them. That is, the internal data structure it uses to store your string when you create one in a Ruby program.

These are represented in Ruby through the RString and RArray structs. The struct definitions of each of these are well commented and have some handy info to help understand them better. What we need to know for our purposes here though is two fold:

RString and RArray have optimizations where for small strings/arrays the values of those strings/arrays will be stored in the struct directly rather than using a pointer to a separate block of memory (embed / ary below rather than ptr). As we’ll see further below, the filenames used in this example fall under that size limit. But if you’re replicating this and have a filename of sufficient length the actual content of the string may be in a slightly different location.
All of Ruby’s internal object representations include a struct called RBasic which holds information about the class it represents and some flags. In effect, internally Ruby objects look something like the following diagram:

With that said, let’s just jump into getting the filenames:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Relative path:
(gdb) call (char*)(
  (struct RString*)(
    (struct RArray*)(
      ruby_current_ec->cfp+1).iseq->body.location.pathobj
    ).as.ary[0]
  ).as.embed.ary
$12 = 0x7faedac1cf50 "sleep.rb"

# Absolute path:
(gdb) call (char*)(
  (struct RString*)(
    (struct RArray*)(
      ruby_current_ec->cfp+1).iseq->body.location.pathobj
    ).as.ary[1]
  ).as.embed.ary
$13 = 0x7faed9f5a0d8 "/tmp/sleep.rb"

And there’s our filename! But, whew, that’s quite the reach into a data structure there. Let’s break that down.

First, we get the pathobj pointer from the iseq location struct with ruby_current_ec->cfp+1).iseq->body.location.pathobj. This part should be fairly familiar by now.
Next, we know from the Ruby source comment that this pointer is either an RString or an RArray so we need to tell GDB how to interpret the memory it points to. In this case, it’s an RArray.
Then, as mentioned above, the array values are small enough to be embedded in the RArray struct directly so the as.ary[0] field is read. This is another pointer; this time to an RString.
Lastly, we again need to cast it to an RString and then we can read its embedded value, as.embed.ary. One more cast to a plain ‘ole char* gets the final string value of the source’s filename.

Before wrapping up, let’s take a quick look at the representation of the RArray and RString values to see how they compare to the diagram of these structs from above:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# RArray:
(gdb) p/x *((struct RArray *)(ruby_current_ec->cfp+1).iseq->body.location.pathobj)
$14 = {
  basic = {
    flags = 0xa00012807,
    klass = 0x7f42be3514b8
  },
  as = {
    heap = {
      len = 0x7f42bee9ce80,
      aux = {
        capa = 0x7f42b9a4a008,
        shared_root = 0x7f42b9a4a008
      },
      ptr = 0x0
    },
    ary = {0x7f42bee9ce80}
  }
}

# RString:
(gdb) p/x *((struct RString*)((struct RArray*)(ruby_current_ec->cfp+1).iseq->body.location.pathobj).as.ary[0])
$15 = {
  basic = {
    flags = 0xa20500805,
    klass = 0x7f42be35ed98
  },
  as = {
    heap = {
      len = 0xe,
      ptr = 0x6c732f656e616873,
      aux = {
        capa = 0x62722e706565,
        shared = 0x62722e706565
      }
    },
    embed = {
      len = 0xe,
      ary = {0x73}
    }
  }
}

The two structs here are fairly similar with the embedded ary and embed fields. If the value is too large for those then the heap field is used instead. Of course, the values in here are mostly pointers or junk values, but nevertheless it’s an interesting look into the internal data structures that back your strings and arrays when using them in Ruby-land.

There are endless more rabbit holes to go down for further exploration into Ruby’s internals here, but at this point we’ve accomplished the goal of peering inside of a running Ruby process from GDB to see what it is executing which makes for a good stopping point. This is barely scratching the surface of how Ruby executes your code. If this topic is interesting and you’re looking for further reading, Pat Shaughnessy’s book Ruby Under a Microscope does an excellent job of covering all of the nitty gritty details on Ruby’s implementation. It targets Ruby 1.9 & 2.0 which are fairly dated now, but the core concepts remain the same making it definitely still worth the read.

Five of the best fly-in camping airports in Washington

Tue, 26 Sep 2023 00:00:00 -0700

Airplane camping in Washington is a far cry from the meccas of Idaho, Utah, or Alaska. Despite all of the forested and mountainous terrain in Washington there are no true backcountry airports in the state. Nevertheless, there are a healthy amount of fly-in camping options on the more beaten path for pilots to enjoy. The tradeoff for this accessibility being, unlike the aforementioned backcountry playgrounds like Idaho, the best airport campgrounds in Washington don’t require planes capable of 500ft takeoff rolls and tundra tires.

The airports listed here are (arguably) the best camping spots in Washington. They are all public airports and have long and smooth enough runways for those flying trikes to operate from. All the while still being some incredible destinations that make flying GA worth all that time, effort, and money. Although if you’re looking for something marginally more challenging, take a look at my most accessible Idaho backcountry airports list.

First, a word of caution: Some of the airports listed here require mountain flying experience and close attention paid to density altitude on those hot summer days. Without mountain flying instruction you don’t know what you don’t know. Before attempting to land at any mountain airport it’s critical that you know your performance numbers, the weather, your approach, your go-around procedure, and everything/anything else you need for operating in the mountains (which you got formal instruction for, right?). There is no guarantee that you can safely land/takeoff from the airports listed here, that is for you to do more research on and ultimately determine.

1. 09S - Sullivan Lake State

Elevation: 2,614ft
Runway length: 1,765ft

At the top of the list is the eastern-most airport on this list. Located in the northeast corner of Washington within the Colville National Forest sits the Sullivan Lake airport. It is in the middle of a national forest campground and directly abuts its namesake, Sullivan Lake.

The campground itself is quite popular and can be difficult to get a campsite reservation at, but for pilots flying in, camping next to your plane is available all along the western side of the airport fence meaning there is never a shortage of space. The campsites here have a handful of picnic tables and firepits. There are also carts for hauling gear should you decide to get a campsite reservation in the campground. The tradeoff for the flexibility of camping inside the fence is that it provides no tree shade. If that’s desirable then getting a campsite in the campground may be worth it.

Once on the ground, the main attraction is the lake. Notably, Sullivan Lake is exceptionally warm making it an excellent swimming location. Located on the beach south of the runway is a swimming dock and further to the east is a floating swimming platform. Sometimes there are buoys for tying off a raft to for prolonged lounging in the lake without being pushed back to shore. Aside from swimming, there are a few hiking trails in the area, one along the east shore of the lake. At night, take advantage of the lack of light pollution to enjoy the plethora of stars on a clear night.

Take note though: The runway here is quite short for your average GA plane, especially one loaded down with camping gear and on hot summer days. At just over 1,700ft you’ll need to have your approach dialed in as there is little margin for error. What’s more, the terrain to the north is rising meaning a late go around is likely not possible and has been the source of crashes in the past. Due to this, landing 34 and departing 16 is the strongly preferred option. When departing, the runway abuts the lake so despite the short field there are no obstacles to clear making for much more comfortable takeoffs than other would otherwise be possible with a runway of that length in your standard GA plane.

Also note that despite fencing running along both sides of the runway, its location between two sides of a campground means it gets a large amount of pedestrian traffic. Doing a low pass before landing is highly recommended to scare off any children that found the open ground a good place to play.

For more info and a detailed map of the airport see Sullivan Lake on Pirep.

^{Credit: Airport map screenshot from Pirep.io}

2. 6S9 - Stehekin State

Elevation: 1,230ft
Runway length: 2,630ft

Stehekin State is the closest we have in Washington to a true “backcountry” airport. The town of Stehekin is located on the north end of Lake Chelan with the only access being by boat, plane, or foot via hiking in. Due to this isolation, Stehekin is generally a quiet outpost in the center of the Cascades.

While camping is not available directly at the airport in Stehekin, the Harlequin Campground (part of North Cascades National Park) is located a short walk from the runway, and is along the shore of the Stehekin River making for an picturesque camping location. Note though that reservations for this campground are required through recreation.gov during the summer. There are eight campsites, but one is first come, first serve if you get there early enough to grab it on a given day.

Within the Stehekin valley there are numerous activities. The airport itself is located about three miles from the lake. If you have folding bikes for your plane, this is a great location for them. Alternatively, for a small fee the Stehekin Shuttle runs up and down the valley road multiple times per day during the summer with a stop near the airport at the Harlequin Bridge.

For those not looking for tent camping, there are commercial outfits in the valley such as the Stehekin Valley Ranch with cabins/rooms for rent and prepared meals.

As for day activities, being inside the national park, excellent hiking options are abound. An easy option near(-ish) the airport is Rainbow Falls (pictured above). For getting on the water, kayak rentals are available near the ferry dock as well as are a few swimming areas.

Last, but far from least, visiting Stehekin necessitates a stop at the bakery. Formally called the Stehekin Pastry Company, all sorts of delicious baked goods are available. If there’s one place to stop in Stehekin, this is it.

With regards to flying in/out, while the elevation of the field is fairly low, density altitude problems do exist on hot summer days. This is compounded by large trees on either end of the runway. The runway is sloped upward to the north as well making arriving 31 and departing 13 the preferred direction. The terrain around here is not forgiving with narrow and dead end canyons in nearly every direction other than south along the lake. Ensure you are proficient with canyon flying and know the terrain well.

For more info and a detailed map of the airport see Stehekin State on Pirep.

3. KORS - Eastsound / Orcas Island

Elevation: 34ft
Runway length: 2,901ft

Eastsound on Orcas Island is a favorite past-time of Puget Sound area pilots. Not only does it make a great day trip location due to its proximity to the town of Eastsound with myriad restaurants, shops, and activities, but it is also a favorite camping location due to the on-field camping, aforementioned attractions, and ease of access.

The camping area on the field is located on the east side in the grass area north of the main ramp. To the east near the hangars is a small bathrooms & showers building. As of this writing, the nightly tie-down fee for camping on the field is $20.

What makes Eastsound shine is not necessarily the on-field camping, but the attractions nearby. The town of Eastsound is a 15 minute walk to the south on a dedicated walking path through the woods. From there, numerous breakfast, lunch, and dinner restaurants are available in addition to boutique shops and a bakery, ice cream shop, and chocolate shop. If the tides are right it’s also possible to walk out to Indian Island.

Aside from options in town, the marina adjacent to the airport has fishing tours, dinner boat tours, kayak rentals, and whalewatching trips. There’s also a twice daily ferry to Sucia Island to the north.

Beyond that, there are too many other activities on Orcas Island to enumerate. Many of these are not within walking distance of the airport, but a few car rental services exist on the island for extended trips. For short trips, there is a crew car available from the airport.

For more info and a detailed map of the airport see Eastsound on Pirep.

4. 4S6 - Tieton State

Elevation: 2,964ft
Runway length: 2,509ft

Located to the southeast of Mt. Rainier sits Rimrock Lake. This large reservoir is frequented by boaters and those staying at the lakeside resorts. On the southeast corner of the lake you’ll find the Tieton State airport. The runway itself abuts the lake making for easy departures to the west without obstacles. However, to the east are many tall trees and rising terrain. Go-arounds are possible, but a late go-around may be extremely difficult on hot summer days. Watch those density altitudes here.

Being a reservoir, the level of the lake will decrease throughout the summer which will affect how close the runway is to the actual water. If you are hoping to hit the water in some capacity it’s worth checking the reservoir level before departing as later in the summer/early fall the lake level may be under 25% full making water access more difficult. See the Bureau of Reclamation’s website for the current fill level. Also note that the lake is filled by snowmelt from the Cascades making it quite cold, generally too cold to swim in comfortably. Kayaking is certainly possible, however.

There are numerous camping options along the field. There is room at the west end of the runway for parking with campsites in the trees near the lakeshore. This area is close to the car camping area, however, and as such may be crowded and loud on nice weekends. The quieter option is the parking area at midfield. This area is well separated from the car camping areas making it more private and only used by pilots.

Speaking of car camping, this is another airport that doing a low pass over before landing would be an excellent idea to scare off any people and animals on the runway. The runway itself is not fenced in and despite some “stay clear” signs, car campers often see fit to drive across the runway. This area is also a range area meaning it’s possible, or even likely, to run into cows grazing on the runway. You’ll potentially also find their cow pies littering the runway providing strong motivation to wash your plane when you get home should you hit one at speed on landing.

For more info and a detailed map of the airport see Tieton State on Pirep.

5. 3W7 - Electric City / Grand Coulee Dam

Elevation: 1,593ft
Runway length: 4,203ft

As the least trafficked location on this list, Electric City located in central Washington provides a quiet camping option. The Electric City airport is located well outside of town and the camp sites are inside the fence so there’s little possibility of crowds other than another pilot there for the fly-in camping as well. The runway itself is fairly straightforward: 4,200ft, paved, and with easy approaches over the lake on both ends.

The campsites are located on the west side of the runway across from the ramp and pilot’s lounge. This grassy area is smooth enough to taxi onto directly obviating the need to park on the ramp and carry your gear down to the campsites (although that’s an option as well). On the ramp you’ll find a pilot’s lounge with bathrooms and drinks/snacks for purchase.

As mentioned, the airport is located outside of town and there’s nothing in the way of attractions around it, but the airport does have a courtesy car for getting into town for food or other activities. The namesake of the town is, of course, the Grand Coulee Dam which is worth a visit. In the summer a nightly laser show is displayed on the face of the dam. See the Bureau of Reclamation’s website for start times as they vary throughout the summer.

For more info and a detailed map of the airport see Electric City on Pirep.

Bonus Airport: 3S7 - Nehalem Bay State / Manzanita

Elevation: 30ft
Runway length: 2,350ft

Finally, as a bonus option to consider, Nehalem Bay is in northern Oregon, but well worth the trip down from Washington. Located just outside the town of Manzanita, Nehalem Bay is a state park that sits along the ocean coast down the Nehalem Spit. The airport itself is a half mile inland, but the exchange for not being immediately on the beach is the quiet camping location inside the airport as opposed to the much busier RV camping areas nearer to the beach.

Sitting at sea level, the paved 2,350ft runway poses few difficulties operating from, especially since the south end abuts Nehalem Bay meaning no obstacles to clear. However, being on the coast means frequent marine layers that prevent morning departures. Plan accordingly, even when the forecast calls for sunny weather.

On the west side of the runway are a handful of campsites nestled in the trees for pilots. These sites carry a $11/night fee for camping on the field as of this writing. The beach is a short half mile walk due west. A paved bike/walking trail runs around the airport as well. This is accessible from the south end of the runway or to the north along the main road.

Aside from the beach and camping, the town of Manzanita is 1.5 miles from the airport. This is walkable, but folding bikes are ideal. Within town many options for food, drinks, and shopping exist. Too many to list here, but a personal favorite being Marzano’s Pizza Pie (protip: be in line at 4pm when they open for the evening).

For more info and a detailed map of the airport see Nehalem Bay State on Pirep.

Shameless Plug

The information on this page came from Pirep, a collaborative database of all public, private, and unmapped airports in the US that all pilots can contribute to, no registration required. Do you have information about other airports in Washington? Or even just local knowledge on your home field? If so, you can help out other pilots learn of new places to fly by taking a few minutes to document it on Pirep.io.

The feasibility of electrifying Mt. Baker

Sat, 26 Aug 2023 00:00:00 -0700

On those rare clear winter days, waiting in the morning lift line for Chair 1 to open at Heather Meadows provides a beautiful view of Mt. Shuksan. Before opening time at 9am it’s common for the lift to start and stop multiple times as the lifties prepare for the day’s operations and ski patrol heads up the mountain to complete avalanche control. Each time the lift starts again the engine revs up providing a guilty reminder that, while other ski areas in the Washington Cascades are electrified, Mt. Baker’s are all powered by burning diesel fuel. The juxtaposition of a frozen Mt. Shuksan looming over the ski area with a diesel engine belching blackened exhaust into the air feels odd at best. Enough so that it made me wonder, what would it take to electrify Mt. Baker? Is it feasible? And should it be done?

Why is this necessary?

First, a disclaimer: I am not an electrical engineer, ski area manager, nor a chairlift mechanic. All of the numbers here are rough, ballpark estimates made from research into approximate figures found from various sources. Sources are cited where needed/available, but please email me if you find an inaccuracy so that it may be corrected.

Why does Baker use diesel to power all of its lifts when other ski areas in Washington use electric? After all, combustion engines are more expensive to operate and less reliable. Well, this is not by choice. It’s because unlike the other ski areas in Washington, Mt. Baker is completely off-grid. So not only are the lifts diesel powered, but also the day lodges, patrol huts, maintenance buildings, and on-site employee housing. The closest power lines are 10 miles away and, as we’ll see in this post, extending those to Baker is quite the expensive endeavor.

There is one caveat to that though: Lift nerds like myself may have noticed that Chair 7 is electric. Indeed, in 2007, the Riblet Chair 7 was replaced with a new electric Skytrac lift. This project included a new diesel generator for powering both the new Chair 7 and all of the White Salmon base area. Supposedly Baker is working on undertaking a similar project for the Raven Hut and chairs 3/4, 5, and 6 by converting their powerplants to electric engines run from a generator as well. Regardless of the actual motor on the chairlift though, every watt of power used at Baker is generated from diesel fuel at some point be it directly from a diesel engine or diesel generator.

But why does this even matter? Given the elevation Baker sits at and having no ability to expand climate change should be of paramount concern for Baker in the coming decades. Contributing to that problem directly by running diesel chairlifts when viable electric options exist would seem counterproductive to the goal of eliminating sources of emissions. More immediately though, programs such as Washington’s cap-and-invest program on emissions will make it more expensive for businesses to emit greenhouse gases (granted, I don’t believe Baker is affected by this particular law because it is below the 25,000 tons emissions limit and sits on federal land within a National Forest, but a not-so-distance future program may apply to Baker). But even if you don’t care about limiting emissions electric engines are more reliable and energy efficient. This means less chairlift downtime and lower operating costs equating to at least marginally slower rising lift tickets prices.

With that in mind, let’s take a look at approximately what Baker’s emissions are, what it would take to electrify it, and if there’s any other practical alternatives.

How much CO₂ does Baker emit?

Naturally the first question to ask about all of this is how much CO₂ is Baker currently emitting per year? We’ll have to make a few assumptions, but this is something that can be approximated to within a ballpark figure using a few base values. Keep in mind these numbers are all rough estimates. The goal is to get within an order of magnitude to the actual number for the purposes of this analysis and are essentially back of the envelope math.

For simplicity, let’s ignore the electricity requirements of the day lodges and operations buildings and focus on calculating the emissions of a single lift then scale that by seven to account for all of Baker’s lifts (Chair 3 & 4 are one lift and accordingly only have a single engine). Doing so will give an average number that should account for differences between length, vertical rise, load factor, and engine models.

From Liftblog, who cites Doppelmayr’s Worldbook (the lift’s manufacturer), we know that Chair 5 uses a 425 Caterpillar engine.

We also know from a 2012 interview with Duncan Howat (the then general manager of Baker) that they use what’s known as a Tier 4 engine. These engines being the most fuel efficient and in turn having the lowest emissions. The closest match I can find for a 425HP, tier 4 engine is the Caterpillar C9.3B engine. At 456HP it’s not a perfect match, but this should be close enough for close enough for our estimation purposes here.

Unfortunately, Caterpillar does not publish fuel consumption or fuel efficiency ratings for their engines, or at least they’re not included on the engine spec sheet. In order to determine the fuel efficiency of an engine we need to know the specific fuel consumption (SFC) of that engine under different loads. In lieu of that graph, we’ll need to make an assumption on that value here. After poking around a handful of research papers, it would seem that a reasonable value for SFC of a diesel engine in the 400HP range is 0.3kg/kWh.¹²

The next question is what is the load factor on this engine? How hard the engine is being run will directly correspond to how much fuel it consumes and thus how much CO₂ it emits. Obviously it’s not being redlined all day long, but it’s difficult to know exactly how hard it is being run so we’ll need to make another assumption here. 425HP is fairly beefy for a fixed grip quad so I believe the load factor would be somewhat low. Let’s say it’s 50% to overestimate.

Now let’s convert the 456HP to kW (since SFC is measured in kg/kWh) and divide by half to account for the 50% load factor. 1HP equals 0.75kW so 456HP * 0.75 * 0.5 = 171kW. From here we can get fuel consumption by multiplying the SFC by the power output which would be 171kW * 0.3kg/kWh = 51.3kg/h. Converting this to imperial units yields 16.1gal/hr given 7lbs per gallon of diesel.

From here, we simply need to multiply by the number of hours per year the lift is running. Baker typically opens late November or early December and closes late April. Let’s assume a good season at 180 days. Hours of operation are 9am - 3.30pm, but let’s call this 8am - 4pm since there are opening/closing tasks that employees use the chairlifts for outside of operating hours. That gives us 8hrs/day * 180 days * 16.1gal/hr = 23,184gal/season. And let’s multiply that by seven for each chairlift (again, Chair 3 & 4 are one lift) to get 162,288gal/season.

When burned one gallon of diesel produces 22.4lbs of CO₂³. That means per season Baker emits (22.4lbs * 162,288gal / 2,000lbs/ton) 1,817 tons of CO₂.

Uh okay, but is that a lot? The EPA estimates the average passenger vehicle emits 4.94 tons per year⁴. That means Baker’s chairlift emissions are equal to 367 passenger cars’ yearly emissions. For reference, that’s roughly equal to the capacity of the back parking lot at Heather Meadows. Given that there are around 5.5 million gas powered vehicles registered in Washington⁵, no, I wouldn’t say 367 is a material number in terms of contribution to state-wide emissions.

Keep in mind that the above calculations are an estimate only and depend considerably on two key assumptions: the specific fuel consumption and load factor of the engine. If the engines are more efficient or run at an average lower load factor then the actual emissions number would be lower, or vice versa if higher.

Overall though, this is less than I thought it would be when I started this estimation. There are some key emissions missing from the estimate though. Namely the snowcats for grooming, but also snowmobiles as well as any trucking of materials up the mountain, which I’ll touch on later. Regardless, “it’s less than I thought” is not zero. And everyone thinking my emissions are only a tiny fraction of a percent is not what is going to get us to zero emissions. So let’s take a look at what it would take to electrify Baker anyway.

Electrification feasibility

It’s worth stating up front that electrifying Baker is a huge undertaking and one that is not going to be cost effective, especially for such a small ski area. It would also involve disruptive construction and logging through miles of a national forest. However, this is exactly the type of project that we need to start taking seriously if emissions targets are to be met.

As a short digression, I write the above because all the renewable energy projects in the world won’t help us if we don’t have a way to transmit the power they generate to where it’s actually needed. While recent federal legislation has poured investment money into a plethora of renewable projects one of my biggest worries is that we’ll fail to sufficiently build the needed renewable infrastructure because of 1. the red tape around actually getting these projects built and 2. the ability to construct transmission lines to move power around the grid to where it’s needed. Transmission lines in particular are exceedingly difficult to build because by definition they cover long distances and touch hundreds or thousands of land owners along their routes. Each land owner with the ability to slow down or stop a project. In short, America needs a new environmentalism, one that wholeheartedly supports infrastructure projects being built in a reasonable and cost effective time frame. As The Economist puts it, preventing clean-energy projects like this is no way to save the planet, and by extension, ourselves. Electrifying Baker is a perfect example of this: it’s expensive and would be a permitting nightmare, yet at some point, it must be done.

That said, let’s run some numbers on what it would take to get power lines to Baker and then consider the overall economics of it and if there’s any alternatives, such as offsetting emissions some other way.

There are three key questions that we need to answer to generate this estimate:

How far from Baker is the nearest grid connection?
How much electric capacity is needed?
How much would X capacity at Y miles cost to install?

Starting off at the top, Baker is marginally closer to the electrical grid than you may think. Naturally one would assume the nearest grid connection is in Glacier. That’s mostly true, but power lines run up the valley another five miles past Glacier due to the Nooksack Hydroelectric Plant between the Church Mountain forest road and Excelsior Pass trailhead. At 2.5MW this isn’t a large power plant (about equal to a single wind turbine in terms of output), but the lines running here are 55kV, which as I’ll get into below, is great for this case.

^{Map source: U.S. Electric Power Transmission Lines}

Assuming it’s possible to extend the power lines here and following a path that closely parallels Route 542, that would mean it’s just about 10 miles to the Raven Hut base area. Ideally you would also need branches to White Salmon, Heather Meadows, and Chair 8 to get everything connected, but I’m going with only Raven Hut here for simplicity and since that’s the most bang for the buck in terms of three out of seven lifts being located there. The most direct line would be ~8 miles, but this 1. crosses over Mt. Herman and 2. would be cutting through a wilderness area so that’s a no-go.

The second question is how much electric capacity is needed to run Baker as the capacity will drive a large portion of the costs. We could do an estimate for approximately how much electric Baker would need if all the lifts were electric and the lodges were included too, but at the end of the day what we’re interested in is how much capacity do these proposed power lines need to have?

In order to get an idea of that we can simply look at other ski areas in the region which are connected to the grid already. Crystal Mountain is a great example here. Puget Sound Energy maintains a buried 34kV line from Greenwater to the Crystal Mountain Blvd. turnoff and then an overhead line to the base area.

Additionally, Crystal also has a small, on-site power generation facility operated by Puget Sound Energy. At 2.5MW it’s not huge and appears to also be diesel generators that only run infrequently in the winter. I’m not sure if this supplemental power or backup power to keep the lights on in the event of an outage. But the lines coming out of it and terminating at the base area are also 34kV. If 34kV is sufficient for Crystal then surely it would be sufficient for Baker as well.

For reference, 34kV lines are fairly small as far as high-voltage lines go. These aren’t the massive steel structures you may think of when hearing “high-voltage transmission lines” (like those that run through the backside of Stevens Pass). To the contrary, 34kV lines have wooden poles and look quite similar to the distribution lines in residential neighborhoods as shown in the diagram below. In terms of effect on the scenery along Route 542, these would be well hidden by the trees. After all, have you noticed the existing 55kV lines leading up to the Nooksack Hydro Plant? Probably not.

It’s worth mentioning here too that relying on overhead power lines through a forest, in the winter, is not exactly a great idea due to frequency of outages during storms. Baker would need some type of on-site backup system in place which would increase costs further, but we’ll ignore that here, again, for the sake of simplicity (alternatively the existing diesel generators could be used and even electric lifts will have a backup diesel engine to evacuate riders in case of power outages or if the electric engine fails). Buried lines would be ideal, but they are far more expensive than overhead lines so for the sake of making this estimate remotely economically feasible only overhead lines are considered.

Given how custom and variable each transmission line project is, it’s difficult to find numbers on what these projects typically cost. The cost would vary greatly depending on location-specific factors. That said, I did find one source from the Public Service Commission of Wisconsin that claimed $285,000 for an overhead 69kV line.⁶ This was in 2011 so inflation adjusted that’s $387,000/mile. It’s also worth noting that Wisconsin is fairly flat, at least compared to the Cascades. So even though this figure is for a 69kV line when we’re considering 34kV lines, I would believe the challenges the terrain around Baker provides more than makes up the price difference between a 34kV line and a 69kV one.

Overall then, at 10 miles and $387,000/mile we’d be looking at somewhere around $3.87M to get power lines to Baker. There are more costs than just the lines though. Some type of substation would be needed to step down the voltage from 34kV to a voltage usable for the lodges and chairlift motors. And you’d also need to run electrical within Baker to reach White Salmon, Heather Meadows, and Chair 8. Not to mention all of the chairlifts (sans Chair 7) would need to actually be converted from diesel to electric as well. Accordingly, $3.87M should be considered an absolute floor on cost.

Is it worth it?

So then is it worth it purely from a financial point of view? Actually, maybe! If Baker is really using ~162k gallons of diesel each year then assuming they’re using dyed diesel (gas tax exempt for off-road use) then it’s likely around $4/gal with prices as of this writing which is around $648k/yr in fuel costs, not counting the costs to truck it up to the mountain. That would mean a breakeven period of just over five years from investing in power lines. However, grid electricity isn’t free either so in reality it would be longer based on the electric costs. In reality, I’d believe that the true cost of running electric lines to Baker would be considerably higher and thus the breakeven period all costs considered could be a decade or more. Is that cost effective then? Eh, no, probably not, but at least it’s a somewhat reasonable timeframe for a return on investment rather than something absurd like 100+ years.

If we’re looking at costs from the lens of emissions reductions, then maybe some government loan or grant could help reduce costs, but I’m purely speculating on that. This is the type of electrification project that likely would not be feasible without government help. On the other hand, there may be other ways to offset emissions. As arguably ineffective and borderline scammy as they are, carbon offsets for all of Baker’s 1,817 tons of CO₂ would cost about $300k/yr.⁷ Or maybe in the future direct air capture will become commercially viable and it would be even cheaper to simply pay to scrub that CO₂ directly from the atmosphere. We’re verging well into the realm of speculation here though.

However, there is one other factor to consider with this. Putting up power lines requires cleared land and when that land is existing forest it involves cutting down trees; trees that are otherwise absorbing CO₂. We need to calculate and subtract the amount of CO₂ those trees absorb each year from the amount Baker emits to get an accurate picture of the net emissions benefit to this type of a project.

Let’s do some more back of the envelope on this. The right of way width for 69kV lines is around 80ft.⁸ We’re talking about 34kV lines, and not all 80ft would necessarily need to be cleared of trees, but this makes for a decent upper bound to estimate off of. So then that’s 80ft * 10 miles = 4,224,000sqft = 96.96 acres.

According to the Washington DNR, the Mt. Baker-Snoqualmie National Forest sequesters 1.83 tons of CO₂ per acre per year.⁹ Given the 96.96 acres figure above, that means we’d lose 177 tons of CO₂ sequestration (in reality a bit less since the land wouldn’t be 100% clear of all vegetation). If Baker emits 1,817 tons/yr then the overall net effect is still ~1,640 tons fewer than continuing to burn diesel.

What about hydrogen?

Baker occupies an interesting area in terms of off-grid energy use. Its energy needs are large enough that we’d like to eliminate the emissions produced fulfilling them with diesel creates, but small enough that it’s not necessarily worth it to run power lines to such a remote location (and also much smaller than something like NuScale is targeting for off-grid use with their SMRs). Given this I believe there’s one other option at least worth taking a look at: hydrogen.

Before getting into it though, hydrogen has its problems as an energy source. It’s expensive to produce, difficult to transport in sufficient quantities, expensive to store, and most current hydrogen production comes from natural gas (so-called “blue hydrogen”) so it’s not doing all that much in the “reduce emissions” department, but, hey, it’s an incremental improvement at least.

Despite the problems, the options for hydrogen electricity generation are growing as the market for it begins to emerge. For example, Caterpillar has generators that run on a blend of natural gas and hydrogen and EODev has a 100% hydrogen generator capable of outputting 110kVA. This is a new enough area that it’s difficult to get numbers on these generators, install costs, fuel sourcing/pricing, and transportation to the mountain in order to do an approximate feasibility analysis. In lieu of that, we’ll have to leave it as “it’s an option on the table,” potentially a quite good one given a few more years of development of hydrogen supply chains.

Also, one notable benefit to on-site generation is its reliability. As mentioned above, overhead power lines carry the risk of outages during storms from falling trees/branches. There’s also the ongoing maintenance and inspection costs associated with miles of lines. That all increases costs even further from the numbers calculated earlier, plus you’d likely need an on-site backup system present anyway. With on-site generation this is simply having spare capacity built-in. Even though on-site generation has the ongoing fuel transportation costs associated with it the overall maintenance costs should be markedly lower and reliability considerably higher.

This is also a good place to consider the use of on-mountain snowcats for grooming and snowmobiles for operations & ski patrol. A small ski area like Baker makes an excellent candidate for electric snowmobiles and electric, or possibly hydrogen powered snowcats. Since these vehicles operate in a fixed area range is not a big concern in that the snowmobiles could be plugged in between use and for grooming, if hydrogen powered, could make refueling trips down the mountain without too much trouble as needed. This is yet another expensive investment, but one that is likely to become more of an option in the not-too-distant future. More analysis is needed here.

Conclusions

Alright, where are we left after all of this? For me, the bottom-line takeaways are:

Baker likely emits somewhere around 1,817 tons of CO₂ per year, or the equivalent of ~367 passenger cars. This is somewhat less than I was expecting when starting this project.
If Baker were to run electric lines to its base area it would result in a net decrease somewhere around 1,640 tons of CO₂ per year. The cost for doing this would be north of $3.87M, likely marginally or considerably higher.

Given those findings, yes, Baker should, ideally, be connected to the power grid, but doing so remains not cost effective. If the funding problem for such a project were solved, then yes, by all means, it should be done. Short of that, however, Baker’s current projects to convert as many lifts as possible to electric powered by diesel generators I believe is the right action to be taking at the present time. This is more efficient financially and ecologically, makes the lifts more reliable, and lays the groundwork for a hopefully future grid-connection electrification project.

On the other hand, it’s difficult to know the future technological, logistical, and economical viability of hydrogen generators but in reality that may prove to be the most realistic option if the mountain is already electrified and the switch becomes as simple as swapping out diesel generators for hydrogen ones. Using the time between now and when that market will hopefully mature to lay the groundwork for such a project by converting the mountain from discrete diesel engines in various locations to centralized diesel generators would appear to be the best move for the time being.

Overall, while it may be disheartening to continue to watch the black chairlist exhaust obscure that view of Shuskan on those clear winter mornings for the foreseeable future, I look forward to the day when we’re riding on electric chairlifts ultimately powered either from clean wind/solar/hydro/nuclear grid power or on-site hydrogen.

Sources

Klanfar, Mario & Korman, Tomislav & Kujundžić, Trpimir. (2016). Fuel consumption and engine load factors of equipment in quarrying of crushed stone. Tehnicki Vjesnik. 23. 163-169. 10.17559/TV-20141027115647. ↩
Saber, H. A., Al-Barwari, R. R. I., & Talabany, Z. J. (2013). Effect of ambient air temperature on specific fuel consumption of naturally aspirated diesel engine. Journal of science and engineering, 1(1), 1-7. ↩
Carbon Dioxide Emissions Coefficients, U.S. Energy Information Administration ↩
Greenhouse Gases Equivalencies Calculator - Calculations and References, U.S. Environmental Protection Agency ↩
Vehicle Registration Counts by State, U.S. Department of Energy ↩
Underground Electric Transmission Lines, Public Service Commission of Wisconsin. (2011) ↩
Terrapass ↩
Transmission Cost Estimation Guide, MISO Energy ↩
Washington Forest Ecosystem Carbon Inventory: 2002-2016, Washington Department of Natural Resources. ↩

Five of the most accessible Idaho backcountry airports

Fri, 28 Apr 2023 00:00:00 -0700

The Idaho backcountry is an intimidating place, and rightly so. Many of the destinations there are unforgiving airports with little to no margin for error. The reward for developing the needed skills to fly here is operating in one of the most scenic and remote areas in the continental US. The most challenging strips are generally not accessible to those without a STOL plane and a good amount of experience, but for pilots looking to dip their toe into the backcountry, there are some options that are accessible to your run of the mill GA plane in addition to a reasonable amount of mountain flying training.

The airports listed here (in no particular order) are some of the more accessible airports in the Idaho backcountry. This means those with longer runways and more forgiving approaches that you don’t need tundra tires and a 500ft takeoff roll to land at. All while still being some incredible destinations that makes flying GA worth it.

First though, a word of caution: Backcountry flying is dangerous and without mountain flying instruction you don’t know what you don’t know. Before attempting to land at any of these airports it’s critical that you know your performance numbers, the weather, your approach, your go-around procedure, and everything/anything else you need for operating in the backcountry (which you got formal instruction for, right?). There is no guarantee that you can safely land/takeoff from the airports listed here, that is for you to do more research on and ultimately determine.

1. 1U1 - Moose Creek

Elevation: 2,454ft
Runway length: 4,100ft

At the top of the list is one of the most remote destinations in the Idaho backcountry. Located next to the Selway river, Moose Creek is a destination to behold. The ranger station there was first established in 1921 with the first runway built in 1931. The second, longer runway was finished in 1958.¹

Today, Moose Creek serves as an excellent camping location. The appeal to Moose Creek is just how remote it is with the nearest road 25 miles away. If you can, visiting on a clear night with a new moon provides some of the least light polluted skies in the continental US.

In addition to camping, being located at the intersection of its namesake, Moose Creek, and the Selway River creates fishing opportunities. Two suspension bridges provide easy crossing of each river. Or if you enjoy hiking, just to the northwest of the field is the trail to the Shissler Peak fire lookout, although with 3,200 vertical feet to the summit.

What makes Moose Creek accessible is its long runway and lower elevation. At only 2,454ft density altitude is less of a problem. But the lower elevation also means marginally warmer temperatures too. The long runway gives departing planes plenty of length to clear the trees at the end, but do mind the terrain around the airport; depending on where and how you land, a go-around may not be possible.

For more info and a detailed map of the airport see Moose Creek on Pirep.

2. 3U2 - Johnson Creek

Elevation: 4,960ft
Runway length: 3,400ft

Johnson Creek is Idaho’s premier backcountry airport. The runway itself is beautifully maintained and the camping facilities are second to no other backcountry airport. If you’re looking for a camping location with all of the amenities, this is it. There are numerous campsites, firepits with firewood available, showers, fresh water, even WiFi access and a refrigerator.

Up the road from Johnson Creek is the town on Yellow Pine with a few food options. One extremely unique part of Johnson Creek is the bathtub hot springs. A short hike from the airport with 400 vertical feet will take you to the hot springs pictured above, complete with an actual bathtub to lounge in while taking in the views of the surrounding mountains.

Johnson Creek is at a decently high elevation of just under 5,000ft often creating density altitude problems. But the long runway makes it accessible to many planes, especially when planning to depart early morning. The valley itself is fairly narrow so knowing your approach is critical. Likewise, traffic here in the summer can be busy at times.

If you visit please consider leaving some money in the donations box to help maintain Johnson Creek.

For more info and a detailed map of the airport see Johnson Creek on Pirep.

3. S81 - Indian Creek

Elevation: 4,718ft
Runway length: 4,650ft

Indian Creek is only 20 miles from Johnson Creek, but it is a true backcountry airport. Located inside the Frank Church - River of No Return Wilderness area, there are accordingly no roads that can take you here.

Indian Creek is a good location for those looking to dip their toes into backcountry flying. The runway is exceptionally long and the approach relatively forgiving. This, plus its location along the Middle Fork Salmon River makes it a popular area for rafters to start their trips from. Aside from rafter drop-offs, it’s generally a fairly quiet airport.

For activities there are numerous hiking trails from the airport. You can hike upsteam to the private Pistol Creek Ranch or downstream along the Middle Fork until you feel like turning back around. Given the remote location you’ll likely feel as if you have a mountain range entirely to yourself.

For more info and a detailed map of the airport see Indian Creek on Pirep.

4. U60 - Big Creek

Elevation: 5,743ft
Runway length: 3,550ft

At a little under 6,000ft, Big Creek is starting to push it for an “accessible” backcountry airport, but the long and downsloped runway makes it possible to still depart first thing in the morning for most planes with a comfortable margin.

Big Creek is home to the Big Creek Lodge. The original lodge was built in 1934 and burned down in 2008. The Idaho Aviation Foundation raised funds to construct a new lodge which opened in 2018. It now again provides breakfast for pilots flying in or lodging for those that are looking to stay overnight. Dinner/lunch is available too with advanced reservations. If not staying in the lodge, a campsite next to the runway is an option too for a nominal fee.

For more info and a detailed map of the airport see Big Creek on Pirep.

5. 0U1 - Warm Springs

Elevation: 4,831ft
Runway length: 2,850ft

Warm Springs is a somewhat unconventional choice for this list. At first glance, the 2,850ft runway and nearly 5,000ft field elevation makes it seem less accessible than it actually is. When taking off to the southwest there are no obstacles and the ground quickly slopes downward. This makes operations considerably easier than needing to climb to clear tall trees.

Otherwise, Warm Springs is a fairly quiet airport. Despite being located off of Route 21, making it not a true “backcountry” airport, it is a good option for those seeking out a quiet place to camp. The main attraction, however, is the hot springs located a short walk down the hill from the runway. The name “Warm Springs” is a bit of a misnomer as the water is near boiling. However, it feeds directly in the Warm Springs Creek where various pools have been constructed of varying water temperatures as the hot and cold water mix allowing you to find the right temperature.

For more info and a detailed map of the airport see Warm Springs on Pirep.

Bonus Airport: 8U2 - Schafer Meadows

Elevation: 4,856ft
Runway length: 3,200ft

Finally, as a bonus option to consider, Schafer Meadows is in Montana, but well worth the visit. Located deep within the Bob Marshall wilderness, Schafer Meadows is a remote backcountry airport. It’s arguably the least “accessible” airport on this list due to its high elevation and semi-short runway with tall trees on both ends.

However, Schafer Meadows is a excellent camping location. The airport campground is located in the trees just to the east of the runway. The ranger station with its herd of horses is worth a walk through as well. River access is available too by following the trail to the east from the campground.

Schafer Meadows receives a good amount of traffic as the start location of many rafting trips on the Middle Fork Flathead river. Come prepared for bugs in the summer too.

For more info and a detailed map of the airport see Schafer Meadows on Pirep.

Shameless Plug

The information on this page came from Pirep.io, a collaborative database of all public, private, and unmapped airports in the US that all pilots can contribute to, no registration required. Do you have information about other airports in Idaho? Or even just local knowledge on your home field? If so, you can help out other pilots learn of new places to fly by taking a few minutes to document it on Pirep.

Sources

Bound for the Backcountry, Richard H. Holm Jr., p405 ↩

The case for reestablishing winter access to Mt. Pilchuck

Mon, 24 Apr 2023 00:00:00 -0700

Last year I wrote extensively about the politics of ski areas in Washington and a vision for the future of skiing in the PNW. One of the topics that came up in both of those posts was Mt. Pilchuck. Namely its history as a ski area and the potential it has for the future. This made me want to explore more on the possibility of reestablishing winter access to Mt. Pilchuck.

First off, for readers who know the history of Pilchuck, this is not a post about re-opening the lift-served ski area that was once the Mt. Pilchuck ski area. The proposal being made here is more modest: specifically reestablishing winter access to the previous ski area parking lot and current summer trailhead. Yup, that’s it. Doing that alone could have a material impact on expanding winter recreational access in the Snohomish county area through expanded high elevation snow access for ski touring, snowshoeing, sledding, and general snow play areas for families.

Mt. Pilchuck History

Before getting into future proposals it’s worth taking a moment to consider the history of Mt. Pilchuck and how we got here.

Sometime between 1913 and 1921 (accounts differ) the fire lookout still on the summit of Pilchuck was constructed.¹² The fire lookouts were, of course, built primarily for fire detection. Given that this is not a concern in the winter, they went unstaffed for those months. There were also far more fire lookouts in addition to forestry/mining cabins at lower elevations. Many of these do not exist anymore due to natural decay after being abandoned or intentional destruction due to wilderness area regulations. However, in the early days this network of primitive cabins allowed ski touring parties to use the cabins/lookouts for extended winter expeditions throughout the Cascades. In the case of Pilchuck, the first known ski descent of the mountain was made in April 1933 by a group of Everett Mountaineers having noted the area for it’s suburb skiing at the upper elevations of the mountain.¹

Fast forward to the 1950s when a logging road was built to the Cedar Flats area on the north side of Mt. Pilchuck. This access allowed for the Mt. Pilchuck ski area to be established in 1956 with a single tow rope.³ Throughout the next decade the Pilchuck ski area added two chairlifts with lights for night skiing, more tow ropes, and a day lodge. In fact, at the time Pilchuck had one of the largest vertical drops of all ski areas in the PNW.

_{Pilchuck in operation}

The trouble with Pilchuck, however, proved to be its elevation. With a base area elevation of only 3,100ft, and one chairlift that descended from there, it would often be raining where other ski areas would be snowing. Some years this was not a problem, like in the 1963-1964 season when the top of the chairlift had a snowpack of a whooping 52ft,⁴ but in other seasons such as the worst snow year on record for the Cascades, 1977-1978, the ski area was only open for three weeks. This year also happened to be its final year in operation.

So why then is there any potential for Pilchuck as a quality skiing location if the snow is this inconsistent and the ski area was forced to close?

Well, there is a caveat to this story: chairlifts never reached the summit of Pilchuck. There was additional, higher elevation terrain that the ski area could have, and wanted to, expand into. Moreover, digging deeper into why it closed we start to see where the politics came into play, which arguably were the primary reason for its closure.

I go more into detail on my post on the politics of ski areas, but in short Pilchuck was closed mostly because of bureaucracy rather than lack of snow. Pilchuck is unique in the Cascades in that it sits partly on both state and federal land. Consider the map below from Caltopo. The green areas (the road and first quarter of the summer trail) sit within the Mount Baker-Sqnoqualmie National Forest and is subject to the whims of the federal Forest Service. Whereas in the purple area, the rest of the summer trail to the summit sit within Washington’s Mount Pilchuck State Park.

What this meant for the ski area was permitting from both the USFS and the Washington State Parks Commission in order to operate. After the poor snow year of 77-78, the ski area knew they had to expand into the higher elevations in order to survive and attempted to do so. However, this would have been on the USFS land and the Forest Service denied the expansion with Joe Nadolski of the Forest Service stating in 1979:

It’s not that we don’t want to see skiing up there. It’s just that we haven’t seen much skiing there since the area opened. That’s the major reason for rejecting the lease renewal and expansion proposal. It’s a low altitude area and it’s often that there’s no snow. We weren’t responsible for Pilchuck’s closure the past two seasons; the weather did them in.⁵

In my opinion, this was an extremely poor and short sighted decision made in the aftermath of literally the worst snow year on record for Washington. It’s commonly said that Pilchuck closed because of lack of snow, but in reality the ski area operators wanted to expand and continue operations, but the Forest Service explicitly prevented them from doing so.

40+ years on as well, we have the snow and the demand for skiing; killing skiing on Pilchuck was a huge mistake. In fact, if Pilchuck had been allowed to expand operations to the summit, and potentially install some snow making capacity at its base, it would be reasonable to assume that it could still be operating today. It would never be as large as the other ski areas in the Cascades, but given its close proximity to the Seattle population it would have an ample supply of skiers looking for quicker trips to the slopes than driving all the way to Stevens.

Today very little evidence of the ski area remains. The day lodge was torn down after years of vandalism. The smaller rentals building was quite literally driven down the mountain circa 1983. Today it sits along the shore of the South Fork Stillaguamish River.

Beyond that, the chairlifts were bought by Crystal Mountain and relocated. Some of their footings are still visible along the summer hiking trail today. Otherwise, here in 2023, the only recognizable remains of the the ski area is the parking lot / present day trailhead. Since the closure the forest has reclaimed the area cleared for the chairlifts and ski slopes to the point that it’s nearly indistinguishable from the surrounding forest. In the photo below the red square denotes the parking lot. The chairlift would have extended perpendicularly up and down from there.

_{This photo was taken on May 16th, 2021. Don’t let the seemingly lack of snow imply this is typical of a mid-winter snowpack here. Even this time of year the top 1,500ft is still excellent skiing.}

Current Access

In the years since the ski area closed, the area has mainly been left unmaintained. The trails receive more or less the same level of attention as any other hiking trail in the Cascades does. The road itself is in an absolutely atrocious condition (more on this later), but most notably for the purposes of skiing, the road to the former ski area parking lot is closed in the winter with it being gated closed at the Heather Lake trailhead.

In terms of ski touring access this closure is especially problematic. The Heather Lake trailhead is located at only 1,300ft whereas the summer trailhead is at 3,100ft. That’s 1,800 vertical feet to effectively get to the start of your tour. To the summit would be nearly exactly 4,000ft. And while we would all love to be able to bang out 4,000ft+ days (assuming you want to get more than one run in for all this work), for the non-diehard skiers that may prefer to do more relaxed days under 4,000ft/day, getting to the summer trailhead is nearly half of your day alone and in conditions that are almost certain to be marginal at best.

Moreover, by horizontal distance walking the road is 5.5 miles to the summer trailhead from the gate. While it’s possible to cut some switchbacks and tour up the former ski area cut in the forest to reduce this distance, this becomes increasingly difficult as the forest continues to grow in and in times with less snow it may not be possible to skin this lower elevation portion.

Given that the potential allure of Pilchuck is close and easy access to high(-ish) elevation snow, the entry tax of just getting to the summer trailhead means it’s usually more worthwhile to visit other locations with easier access instead. In years and decades past when we had far fewer people visiting our mountains in the winter that was fine, but the entire point of this article is that this road closure creates a wasted opportunity to better disperse crowds in our otherwise overcrowded winter-time mountains. Let’s take a look at why another snow access point is needed.

Why create more winter recreation access?

As I discussed in a previous post, Washington’s population has grown by 50% since 1990 with another 30% expected by 2050.⁶ Accordingly, this has put stress on recreational access in the Cascades. This is true for summer recreation, but especially so for winter recreation due to the limited number of access points to high elevation snow.

In fact, there are really only seven high elevation snow access points in Western Washington:

With the exception of Paradise, each of these also correspond to a ski area. For ski touring, this has the downside of using the same roads and parking lots as these ski areas when they are already vastly overcapacity themselves.

It’s not only ski touring, however. Other forms of winter recreation including snowshoeing, sledding, and general snow play areas for families with children are also in high demand. Having places to separate these activities from the ski areas would help relieve pressure on the ski areas while also allowing for more opportunities to partake in them.

Why is Pilchuck the right place for this?

So then, why is Pilchuck the right location to create this access?

For starters, Pilchuck is a notably high elevation mountain for being so far on the western edge of the Cascades. This brings the advantage where it’s more easily accessible to North Seattle and Snohomish County residents than many other areas located deeper in the Cascades. Likewise, since it sits along Mountain Loop Highway, rather than a mountain pass, the amount of traffic in the area during the winter is minimal. Compare this to the traffic on US-2 through Sultan which even in the winter can add an hour or more to your trip back to Seattle. Expanding winter access along Mountain Loop Highway makes use of an underutilized area of the Cascades in the winter and disperses a small amount of the would-be US-2 traffic elsewhere.

As for the terrain itself, Pilchuck tops out at 5,344ft. While this isn’t anything crazy high for the Cascades, it is at a good elevation for skiing sitting between the summit elevations of Baker and Stevens. I previously analyzed the Cascade’s snowpack over the past 100 years to analyze how it was responding to climate change. The takeaway being that anything over 4,000ft is likely to remain decent skiing for the foreseeable future. While it’s probably not the best location to be considering for the capital investment required to install chairlifts again, it’s not a massive undertaking to plow an existing road either.

Similarly, the terrain itself is excellent for skiing. Obviously it was good enough for a ski area in the past, but even better terrain exists coming off the top of the mountain. The Gunsight Couloir (featured in the photo at the top of this article) is a 40° slope just to the east of the fire lookout. From the bottom of it, and with sufficient snow, the south face of the mountain can be skied lower down as well before skinning back up to ski the north face back down. In fact, large parts of the northern side of the mountain are open faces leading into sparsely gladed runs. All of this makes Pilchuck a popular touring location currently and, I believe, gives it the potential to be even better with a high elevation trailhead open.

_{The north aspect of Pilchuck; lots of great lines to be had here from the summit towards the trailhead to the west or Larrison Ridge to the east. With sufficient snow there are also chutes that descend into Heather Lake (not pictured here). This photo was also taken in May 2021.}

Another notable feature of Pilchuck is the fire lookout at its summit. Like some other fire lookouts in the Cascades, this could be made available for overnight stays in the winter too making it possible to stay at the higher elevation for multiple days without needing to haul as much gear up with you. This assumes it is properly cared for and maintained, of course. Other lookouts that used to be open in the winter for these purposes have since been closed due to neglect, unfortunately.

And finally, while many mountains in the Cascades have favorable skiing qualities, the primary attribute that sets Pilchuck apart is its road access. It was already mentioned how Pilchuck sits off to the side of Mountain Loop Highway, which is already presently maintained in the winter well past the turnoff to Pilchuck. The forest road itself is open in the winter up to the Heather Lake trailhead. This means that in order to establish winter access to the summer trailhead, all that is needed is a little over five miles of plowing.

Moreover, this road weaves its way through the forest. There are no cuts into the side of the mountain, avalanche chutes, or really any significant danger that would require additional mitigation in the winter beyond routine plowing. And at the end of the road is the former ski area parking lot which is decently sized and able to hold a good number of cars for anyone using it on a given day (as opposed to other trailheads where finding parking can be an adventure of trying to not block the road while getting uncomfortably close to a large drop off). Another advantage this road has for winter access is that the top half of it is actually already paved. This is, of course, due to the former ski area; I don’t know if the bottom half was ever paved as it is dirt currently. It certainly would need repairs in areas, but the mere fact that the higher elevations are paved already is a major advantage to keeping it open in the winter.

As for the bottom dirt half, it is worth discussing the current state of that portion because it is… bad to put it mildly. I really can’t overstate how poor of a condition the lower half of this road is. There are potholes on this road that give Heather Lake a run for its money in terms of depth. It is easily the worst actively used forest road I’ve driven on in the Cascades and to make matters worse it is lined end to end with cars on any given summer weekend around the Heather Lake trailhead making it impossible to dodge the potholes.

I’m far from a snowplowing expert, but my feeling is that before this road could be kept open in the winter there would need to be repairs to at least portions of the paved section and major repairs to the dirt section. However, given the popularity of the Heather Lake trail in the summer, this would be a welcome improvement regardless of if it’s kept open in the winter or not. In reality, it may be best to just pave the entire length given its summer popularity and the ease of plowing it would facilitate.

Economics

As with everything in this world, keeping roads open in the winter costs money. So how would this be paid for?

There are two areas in particular that need funding before the road could be kept open to traffic:

Road repair to improve the lower dirt half and spot repairs to the upper paved half.
Ongoing funding to pay for snowplowing.

Let’s cover the second item first. Assuming the road was repaired enough to make it feasible to plow in the first place, how would said plowing be paid for?

One option, and probably the easiest, would be a volunteer program. Like many forest roads in the Cascades, simply leave the gate open and let adults be responsible for their own decisions about what they can and cannot make it up. From there, the touring community could potentially self-organize volunteer plowing given that this road isn’t an especially difficult one to keep clear with the lack of avalanche problems.

Another option would be a voluntary donation model similar to the Friends of Hyalite program in Montana. In that program, users of the area donate $20/yr to reimburse the county for 60% of their costs to keep the road open. More research is needed on specific numbers for what would be required here, but this model could likely work well in Washington too. Snohomish county may even be interested if it relieves some traffic along US-2. Plus, unlike Montana, the lowlands of Western Washington rarely get snow in the winter. This means that country snow removal resources are generally more available.

Now for road repairs, this is a difficult issue that there’s not a clear solution for as the road itself sits on National Forest land and funding for our forest roads is stretched thin enough as-is with many roads being essentially abandoned due to lack of funds to maintain them. However, while I was writing this article some news about new logging projects in the area has come out which is planned to expand the Heather Lake trailhead parking from 25 cars to 75.⁷ Will this also bring along the needed road repairs? That is to be determined, but I’d say it’s a good chance that if the trailhead is being expanded that the road segments around the trailhead, which are the worst parts, would be repaired as well.

As a side note, I know we all generally dislike logging in the Cascades, especially in areas used for recreation, but this is a good reminder of how the forest roads we all enjoy for access to trailheads were quite literally built on the back of the logging industry of yesteryear. When the areas to be logged are second-growth forest that that has become more dense than old-growth forest and is located near towns causing wildfire concerns I think it’s reasonable to make the tradeoff of some logging in exchange for forest health and recreation funding (especially after the Bolt Creek Fire last year demonstrated the ability for this area of the Cascades to produce devastating fires).

Otherwise excluding this new logging project, unless federal funding is to be provided it’s not likely that the Forest Service would be able to fix the lower half of the road let alone asphalt repairs to the paved half. Maybe the state would be willing to pony up some road repair funding since the road is for accessing its state park, afterall?

There is another option, however; one that I think is particularly interesting. Bluebird Backcountry in Colorado is a ski area, but without chairlifts. For day tickets starting at $40 (or a season pass if you choose) you get access to their avalanche-evaluated ski area with marked trails, ski patrol, rental options, and avalanche courses, which is all only accessible via skinning.

Given the ever growing popularity of ski touring, a similar model in Washington could work excellently here. Having a spot close to Seattle where folks new to touring could rent gear, get instruction, and try out touring in a professionally avalanche-evaluated (not necessarily mitigated) area could be a huge benefit to the sport while also instilling good habits in new folks from the start.

In exchange, part of the revenue from this ski area would be used for road repairs and snowplowing. Plus, by definition, the only infrastructure needed is a small, even temporary building for rentals, tickets, and patrol. Initial set up costs would be minimal compared to that of a lift-accessed ski area. This model provides the funding benefits that a commercial operation provides but without the large capital investment of chairlifts.

Conclusions

Overall, the potential access that Pilchuck could provide for numerous forms of winter recreational in Western Washington is extremely promising. It’s in the right location, has good road access, great ski terrain, permissive land zoning, and a history of skiing. All we need to unlock this potential is winter road maintenance.

So could we make this happen? Last year I wrote a series of articles on the potential and pitfalls of expanding lift-served access in Washington. Most of this being pie-in-the-sky type ideas. Compared to that, re-opening Pilchuck seems like a very down to Earth type of proposal. Since last year I’ve had a handful of people reach out to me regarding ski access in Washington. One of the topics we’ve always come back to is how there is a lack of advocacy groups for skiers here, both backcountry and lift-served. We’ve traditionally relied on ski areas to provide this access for us. However, in the rapidly evolving world of corporate consolidation in the ski industry it likely will end up being in our and the sport’s best interest for a grassroot skiers organization to come together to work with the federal and state agencies to bring proposals like this one to fruition. Are you interested?

Sources

Written in the Snows, Lowell Skoog, p144 ↩ ↩²
To get to iconic Pilchuck lookout, hikers must brave ‘hell on wheels’, The Daily Herald ↩
Written in the Snows, Lowell Skoog, p186 ↩
Mt. Pilchuck Ski Area ↩
The Dilemma at Mt. Pilchuck, Northwest Skier (1979) ↩
Forecast of the State Population, State of Washington ↩
Forest Service wins Stillaguamish logging suit over conservation group, The Daily Herald ↩

Geographically ranked Postgres full text search

Tue, 21 Mar 2023 00:00:00 -0700

Here’s a problem: You have 20,000 records each with latitude and longitude coordinates. You want a search function for these records to show the results closest to the user’s current position on a map. What do you do?

Thanks to Postgres’ full text search and Earthdistance extension we can implement this logic entirely in the database and it’s lightning fast to boot.

This is exactly the problem I had when I was working on my recent project Pirep. The core functionality of the website is based around a map which displays ~20,000 airports in the US provided by the FAA’s database. The map has a search feature on it and I wanted it to display results based on the area the user was looking at on the map. For example, searching for Portland when looking at New England would return the Portland, Maine airport rather than the Portland, Oregon airport even though Oregon would likely be the more common result given it’s a larger city.

As far as searching goes, given a database with all of the airports in it, indexing by airport name and location isn’t difficult. The difficulty comes in when trying to rank search results based on the proximity to the user’s current location. It’s not exactly feasible to read every record, calculate its distance to the user, sort by distance, and then take your search results in realtime. You could conceivably index airports by their state and only return results from the same state as the user then calculate distance on those results and sort by that. That may be a “good enough” solution, but in my case I wanted, for example, a search for port when looking at the south Puget Sound to return Port Orchard, WA at the top of the list instead of the further away Port Angeles, WA. Thanks to the Earthdistance extension it’s possible to do this type of calculation entirely in the database and in realtime.

Searches Schema

First though, let’s cover the basics of Postgres’ full text search in general and how we can use it to index our records and then search and rank them based on our application’s needs. To start off, all of my search records are stored in a searches table with the following schema:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
searches_demo=# \d searches;
                                        Table "public.searches"
     Column      |          Type          | Collation | Nullable |               Default                
-----------------+------------------------+-----------+----------+--------------------------------------
 id              | integer                |           | not null | nextval('searches_id_seq'::regclass)
 searchable_type | character varying(255) |           | not null |
 searchable_id   | integer                |           | not null |
 term_vector     | tsvector               |           | not null |
 term            | character varying(255) |           | not null |

Indexes:
    "searches_pkey" PRIMARY KEY, btree (id)
    "searches_searchable_id_searchable_type_term_idx" UNIQUE, btree (searchable_id, searchable_type, term)
    "searches_searchable_type_searchable_id_idx" btree (searchable_id, searchable_type, term)
    "searches_term_vector_idx" gin (term_vector)

This table’s schema is fairly simple with the following columns:

id: Primary key
searchable_type: The record type that this search record corresponds to.
searchable_id: The record ID that this search record corresponds to.
term_vector: The compiled search term that Postgres will use for matching against a user inputted query.
- The tsvector type is a Postgres type that stores this information and will be used heavily here for search indexing and conducting searches.
term: The raw search term that will match this search record as a search result.

For indicies a GIN index exists for the term vector for quick searching. Additionally, a unique index on searchable ID, searchable type, and the search term exists for doing upsert statements when indexing records.

Indexing

Speaking of search indexing, let’s go into how to populate this table. All of this functionality is used inside of a Rails app so we could do something simple like iterate over all of our records and upsert a search record for each of them. This isn’t very performant though; we could instead do all of the indexing directly in the database nearly instantaneously with an INSERT INTO SELECT statement.

Let’s consider we have an airports table which contains some basic information about airports:

1
2
3
4
5
6
7
8
9
10
11
searches_demo=# \d airports;
                                    Table "public.airports"
 Column |          Type          | Collation | Nullable |               Default                
--------+------------------------+-----------+----------+--------------------------------------
 id     | integer                |           | not null | nextval('airports_id_seq'::regclass)
 code   | character varying(255) |           |          | 
 name   | character varying(255) |           |          |

Indexes:
    "airports_pkey" PRIMARY KEY, btree (id)
    "airports_code_key" UNIQUE CONSTRAINT, btree (code)

We’ll populate it with a few records:

1
2
3
4
5
6
7
searches_demo=# SELECT * FROM airports;
 id | code |                    name                     
----+------+---------------------------------------------
  1 | SEA  | Seattle-Tacoma International
  2 | SFO  | San Francisco International
  3 | BLI  | Bellingham International Airport
  4 | ANC  | Ted Stevens Anchorage International Airport

With that all set up, if we want to create search records for every airport indexing by its name all we need is one query as such:

1
2
3
4
5
6
7
8
9
10
11
12
13
INSERT INTO searches (
  searchable_id,
  searchable_type,
  term_vector,
  term
)
SELECT
  id,
  'Airport',
  to_tsvector('simple', name),
  name
FROM airports
WHERE name IS NOT NULL AND name != '';

Alright, so what’s going on here? Essentially we’re taking every airport record and creating a new search record for it with its ID set to searchable_id, searchable_type set to Airport, and the search term set to the airport name for airports where it’s not null or an empty string.

The important part is the to_tsvector('simple', name) line. This tells Postgres to convert the airport’s name to a tsvector value. This is a sorted list of unique lexemes or tokens which are normalized to represent variants of the words in the input. Basically, it’s a special data structure that indexes our search terms so that when we search for them later Postgres can do its magic to return search results for us.

To demonstrate, we can see below how Postgres will transform a given sentence into a tsvector:

1
2
3
4
# select to_tsvector('simple', 'The quick brown fox jumps over the lazy dog');
                                to_tsvector                                
---------------------------------------------------------------------------
 'brown':3 'dog':9 'fox':4 'jumps':5 'lazy':8 'over':6 'quick':2 'the':1,7

It’s worth pointing out the first argument to this function. Above it’s set to simple which tells Postgres to not perform any mangling of the input. Instead if we set it to english when Postgres will perform some manipulation on the input to adapt for language-specific features for yielding hopefully better search results. Notably, the lexeme the is omitted entirely from the tsvector below since it would not yield relevant search results in most cases.

1
2
3
4
# select to_tsvector('english', 'The quick brown fox jumps over the lazy dog');
                      to_tsvector                      
-------------------------------------------------------
 'brown':3 'dog':9 'fox':4 'jump':5 'lazi':8 'quick':2

Depending on your use case you may not want Postgres to manipulate your input and instead index it as-is. This airport name case is one of those situations. Here we have all proper nouns that Postgres should not be changing. If your search terms involve primarily unique values you’d probably want to use simple as well but if you’re searching more free-form documents then english (or whatever language the content is in) would be appropriate.

Additionally, there are many more options for controlling how Postgres parses your search terms. The Postgres manual has excellent documentation on how to use these.

Getting back to our query, if we run it with the following records in our airports table we get the search table populated as such:

1
2
3
4
5
6
7
searches_demo=# SELECT * FROM searches;
 id | searchable_type | searchable_id |                           term_vector                           |                    term                     
----+-----------------+---------------+-----------------------------------------------------------------+---------------------------------------------
  5 | Airport         |             1 | 'international':4 'seattle':2 'seattle-tacoma':1 'tacoma':3     | Seattle-Tacoma International
  6 | Airport         |             2 | 'francisco':2 'international':3 'san':1                         | San Francisco International
  7 | Airport         |             3 | 'airport':3 'bellingham':1 'international':2                    | Bellingham International Airport
  8 | Airport         |             4 | 'airport':5 'anchorage':3 'international':4 'stevens':2 'ted':1 | Ted Stevens Anchorage International Airport

Searching

With a populated search table we can finally start running some searches. This is as easy as a select statement:

1
2
3
4
5
6
7
searches_demo=# SELECT airports.* FROM "airports"
INNER JOIN searches ON searches.searchable_id = airports.id
WHERE (term_vector @@ to_tsquery('simple', 'seattle'));

 id | code |             name             
----+------+------------------------------
  1 | SEA  | Seattle-Tacoma International

Let’s break this down a little bit. Everything here is a basic select statement with a join and a where clause. The key components are the @@ operator and the to_tsquery function. Here we’re taking our search query, seattle, converting it to a tsquery and then using the @@ operator to tell Postgres to check if there’s a match between the tsquery value and the tsvector values in our search table. The matching rows become our search results so we pull the associated records from the airports table with the join.

Similar to the to_tsvector function, there’s a bunch of additional functionality here. First and foremost, the language argument behaves the same way. In fact, it should match the language the tsvector values were created with or we may not get any results. For example the same query with the language set to english will find nothing:

1
2
3
4
5
6
7
searches_demo=# SELECT airports.* FROM "airports"
INNER JOIN searches ON searches.searchable_id = airports.id
WHERE (term_vector @@ to_tsquery('english', 'seattle'));

 id | code | name 
----+------+------
(0 rows)

Depending on your use case, you may want to specify prefix matching for your queries. Consider the following case where if we search for just inter we get no results:

1
2
3
4
5
6
7
searches_demo=# SELECT airports.* FROM "airports"
INNER JOIN searches ON searches.searchable_id = airports.id
WHERE (term_vector @@ to_tsquery('simple', 'inter'));

 id | code | name 
----+------+------
(0 rows)

But by changing the query to inter:* hence telling Postgres to treat the query as a prefix search we get what would probably be a more expected result:

1
2
3
4
5
6
7
8
9
10
searches_demo=# SELECT airports.* FROM "airports"
INNER JOIN searches ON searches.searchable_id = airports.id
WHERE (term_vector @@ to_tsquery('simple', 'inter:*'));

 id | code |                    name                     
----+------+---------------------------------------------
  1 | SEA  | Seattle-Tacoma International
  2 | SFO  | San Francisco International
  3 | BLI  | Bellingham International Airport
  4 | ANC  | Ted Stevens Anchorage International Airport

In addition to to_tsquery, there are a handful of other conversion functions worth reading about that may be useful depending on your use case. For example, plainto_tsquery, phraseto_tsquery, and websearch_to_tsquery.

Ranking Results

The above queries are simply returning whichever search records matched search terms with the given query; there’s no specified ranking or ordering of search results. If the ultimate goal is to have our search results be in order of what is closest to the user we’ll need a way to rank them. This is where the setweight and ts_rank functions come into play.

In order to get ranked search results we first need to specify what weighting of search records should be. This will depend on the business logic needs of your application but for this case let’s say we want to weight public airports higher than private airports. We’ll add a new facility_use column to the airports table to denote this with either public or private strings values.

1
2
3
4
5
6
7
8
searches_demo=# SELECT * FROM airports;

 id | code |                    name                     | facility_use 
----+------+---------------------------------------------+--------------
  5 | SEA  | Seattle-Tacoma International                | public
  6 | SFO  | San Francisco International                 | private
  7 | ANC  | Ted Stevens Anchorage International Airport | public
  8 | BLI  | Bellingham International Airport            | private

Now we can use this new column when creating search records to apply weighting to the tsvector values. First though we need to understand how Postgres weighs tsvector values. It’s fairly simple actually: Postgres uses the letters A, B, C, and D for weights. The intention behind this is that different parts of a text document carry more weight than others. For example, the title would have weight A and the body have weight D. However, note that the numerical values derived from these weights that will order results are not tied to these letters. For example, it would be natural to assume that a weight of A would correspond to the highest priority, but that’s not necessarily true depending on how your structure your search query. In fact, as you’ll see below, Postgres’ default is that D is the highest weighted value. Think of the letters more as a categorization method rather than strictly a weighting method. For one type of search query you may want lexemes with weight A to be the most relevant and another weight B should be at the top. More on this below.

In our example here, we’ll give public airports a weight of D and private a weight of C since the more common airports will generally be the public ones (again, because by default Postgres will make D the most relevant results). To do this we’ll modify the INSERT statement above that creates the search records to include the weighting with the setweight function.

1
2
3
4
5
6
7
8
9
10
11
12
13
INSERT INTO searches (
  searchable_id,
  searchable_type,
  term_vector,
  term
)
SELECT
  id,
  'Airport',
  CASE WHEN facility_use = 'public' THEN setweight(to_tsvector('simple', name), 'D') ELSE setweight(to_tsvector('simple', name), 'C') END,
  name
FROM airports
WHERE name IS NOT NULL AND name != '';

The key modification being the conditional statement where the to_tsvector calls are wrapped with setweight calls with the relevant D or C arguments. Now after viewing the search records we can see how lexemes in the tsvector values have weights associated with them:

1
2
3
4
5
6
7
8
searches_demo=# SELECT * FROM searches;

 id | searchable_type | searchable_id |                             term_vector                         |                    term                     
----+-----------------+---------------+-----------------------------------------------------------------+---------------------------------------------
 13 | Airport         |             5 | 'international':4 'seattle':2 'seattle-tacoma':1 'tacoma':3     | Seattle-Tacoma International
 14 | Airport         |             6 | 'francisco':2C 'international':3C 'san':1C                      | San Francisco International
 15 | Airport         |             7 | 'airport':5 'anchorage':3 'international':4 'stevens':2 'ted':1 | Ted Stevens Anchorage International Airport
 16 | Airport         |             8 | 'airport':3C 'bellingham':1C 'international':2C                 | Bellingham International Airport

Now we can perform weighted searches against these terms. As mentioned above, the letters themselves don’t assume that D is always the most relevant. When doing a search query we need to apply the ts_rank function to assign numerical values to each letter. By default, Postgres will apply the following weights to the values D through A respectively: {0.1, 0.2, 0.4, 1.0}.

This will control which values carry the most weight and subsequently the most relevant results. It is possible to assign your own custom values too for changing this behavior on the fly. The tsrank function takes an optional argument that let’s you specify your own weight values.

As with most things here, Postgres provides a bunch of configurability to ranking. It’s worth giving the documentation a read through.

With all that in mind, our new search query for the word “international” will look as such:

1
2
3
4
5
6
7
8
9
10
searches_demo=# SELECT airports.*, ts_rank(term_vector, 'international') AS rank FROM airports
INNER JOIN searches ON searches.searchable_id = airports.id
WHERE (term_vector @@ to_tsquery('simple', 'international')) ORDER BY rank ASC;

 id | code |                    name                     | facility_use |    rank    
----+------+---------------------------------------------+--------------+------------
  5 | SEA  | Seattle-Tacoma International                | public       | 0.06079271
  7 | ANC  | Ted Stevens Anchorage International Airport | public       | 0.06079271
  6 | SFO  | San Francisco International                 | private      | 0.12158542
  8 | BLI  | Bellingham International Airport            | private      | 0.12158542

Breaking this down, the search query is nearly the same as before with two notable changes:

The addition of the calculated rank column with ts_rank(, term_vector, 'international') AS rank.
The ORDER BY rank ASC clause to sort by the calculated rank.

As we’d hope for, this places the public airports above the private ones.

Ranking by Distance

All of the above was building up to the point where we start mixing in the Earthdistance extension. Earthdistance is a Postgres extension that allows us to compute great circle distances between coordinate points directly in the database. We can use it in combination with the full text search concepts above to create location-aware search queries. Before using it, it must first be enabled with:

1
CREATE EXTENSION earthdistance CASCADE;

The first task we need to accomplish is adding latitude and longitude location data to the airports. Since we’ll be operating on these values in the database we need to use the correct data type; a point type in this case. Let’s make new airports and searches tables with a coordinates column of type point and insert airports into the table:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
CREATE TABLE airports (
  id SERIAL PRIMARY KEY,
  code VARCHAR(255) UNIQUE,
  name VARCHAR(255),
  coordinates POINT
);

CREATE TABLE searches (
  id SERIAL PRIMARY KEY,
  searchable_type VARCHAR(255) NOT NULL,
  searchable_id INTEGER NOT NULL,
  term_vector TSVECTOR NOT NULL,
  term VARCHAR(255) NOT NULL,
  coordinates POINT
);

INSERT INTO airports (code, name, coordinates) VALUES ('SEA', 'Seattle-Tacoma International', point(-122.31177777, 47.44988888));
INSERT INTO airports (code, name, coordinates) VALUES ('SFO', 'San Francisco International', point(-122.37541666, 37.61880555));
INSERT INTO airports (code, name, coordinates) VALUES ('ANC', 'Ted Stevens Anchorage International Airport', point(-149.9981375, 61.17408472));
INSERT INTO airports (code, name, coordinates) VALUES ('BLI', 'Bellingham International Airport', point(-122.53752777, 48.79269444));

SELECT * FROM airports;
 id | code |                    name                     |         coordinates         
----+------+---------------------------------------------+-----------------------------
  1 | SEA  | Seattle-Tacoma International                | (-122.31177777, 47.44988888)
  2 | SFO  | San Francisco International                 | (-122.37541666, 37.61880555)
  3 | ANC  | Ted Stevens Anchorage International Airport | (-149.9981375, 61.17408472)
  4 | BLI  | Bellingham International Airport            | (-122.53752777, 48.79269444)

INSERT INTO searches (
  searchable_id,
  searchable_type,
  term_vector,
  term,
  coordinates
)
SELECT
  id,
  'Airport',
  to_tsvector('simple', name),
  name,
  coordinates
FROM airports
WHERE name IS NOT NULL AND name != '';

SELECT * FROM searches;
 id | searchable_type | searchable_id |                           term_vector                           |                    term                     |         coordinates         
----+-----------------+---------------+-----------------------------------------------------------------+---------------------------------------------+-----------------------------
  1 | Airport         |             1 | 'international':4 'seattle':2 'seattle-tacoma':1 'tacoma':3     | Seattle-Tacoma International                | (-122.31177777, 47.44988888)
  2 | Airport         |             2 | 'francisco':2 'international':3 'san':1                         | San Francisco International                 | (-122.37541666, 37.61880555)
  3 | Airport         |             3 | 'airport':5 'anchorage':3 'international':4 'stevens':2 'ted':1 | Ted Stevens Anchorage International Airport | (-149.9981375, 61.17408472)
  4 | Airport         |             4 | 'airport':3 'bellingham':1 'international':2                    | Bellingham International Airport            | (-122.53752777, 48.79269444)

Note that the facility_use column was dropped since we’re not using it anymore.

The above should get our database set up. For the fun part, we can now write our search query. Essentially the method boils down to taking the computed rank and further weighting it by distance. This is done by multiplying the rank value by the distance between the user’s given location and the airport’s location.

Because we’re using points the complex part of computing this distance becomes as simple as using the <@> operator. This will compute the distance between two points in statue miles. It assumes that the Earth is a perfect sphere which of course isn’t true, but should be accurate enough for most purposes.

Finally, putting this all together we end up with the following search query (which is looking for airports closest to Anchorage):

1
2
3
4
5
6
7
8
9
10
searches_demo=# SELECT airports.*, ts_rank(term_vector, 'international') * (point(-149.069051, 60.962834) <@> searches.coordinates) AS rank FROM airports
INNER JOIN searches ON searches.searchable_id = airports.id
WHERE (term_vector @@ to_tsquery('simple', 'international')) ORDER BY rank ASC;

 id | code |                    name                     |         coordinates         |        rank        
----+------+---------------------------------------------+-----------------------------+--------------------
  3 | ANC  | Ted Stevens Anchorage International Airport | (-149.9981375,61.17408472)  | 2.0859955802743917
  4 | BLI  | Bellingham International Airport            | (-122.53752777,48.79269444) |  81.03005742401568
  1 | SEA  | Seattle-Tacoma International                | (-122.31177777,47.44988888) |  85.76396945761945
  2 | SFO  | San Francisco International                 | (-122.37541666,37.61880555) | 120.54008426527699

Just like that, we have the search results in order from closest to furthest away. Note that the rank column is proportional in magnitude to the distance from the given location in that the two Washington airports (Seattle and Bellingham) are close in value while San Francisco is still further away. With this, it may be useful to drop off search results entirely based on a rank value if far away results are deemed not relevant.

And of course, this is still a search so if the search term is changed to something that does not match all of the airports, like seattle, then only the relevant search results are returned:

1
2
3
4
5
6
7
searches_demo=# SELECT airports.*, ts_rank(term_vector, 'seattle') * (point(-149.069051, 60.962834) <@> searches.coordinates) AS rank FROM airports
INNER JOIN searches ON searches.searchable_id = airports.id
WHERE (term_vector @@ to_tsquery('simple', 'seattle')) ORDER BY rank ASC;

 id | code |             name             |         coordinates         |       rank        
----+------+------------------------------+-----------------------------+-------------------
  1 | SEA  | Seattle-Tacoma International | (-122.31177777,47.44988888) | 85.76396945761945

Best of all, these queries are lightning fast. In Pirep, the searches table has ~40,000 rows in it since each airport is indexed by both its code and name. Ranking by distance with that table size is near instantaneous:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
pirep_development=# SELECT count(*) FROM searches;

 count 
-------
 41324

pirep_development=# EXPLAIN ANALYZE SELECT airports.code, ts_rank(term_vector, 'intl:*') * (point(-98.57944574225633, 39.82834557323) <@> searches.coordinates) AS rank FROM airports INNER JOIN searches ON searches.searchable_id = airports.id WHERE (term_vector @@ to_tsquery('simple', 'intl:*')) ORDER BY rank ASC;

                                                                         QUERY PLAN                                                                          
-------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=1980.21..1980.94 rows=293 width=12) (actual time=6.216..6.233 rows=273 loops=1)
   Sort Key: ((ts_rank(searches.term_vector, '''intl'':*'::tsquery) * ('(-98.57944574225633, 39.82834557323)'::point <@> searches.coordinates)))
   Sort Method: quicksort  Memory: 37kB
   ->  Hash Join  (cost=540.37..1968.20 rows=293 width=12) (actual time=0.463..6.138 rows=273 loops=1)
         Hash Cond: (airports.id = searches.searchable_id)
         ->  Seq Scan on airports  (cost=0.00..1319.47 rows=20647 width=20) (actual time=0.006..3.769 rows=20647 loops=1)
         ->  Hash  (cost=536.70..536.70 rows=294 width=60) (actual time=0.430..0.430 rows=273 loops=1)
               Buckets: 1024  Batches: 1  Memory Usage: 39kB
               ->  Bitmap Heap Scan on searches  (cost=26.28..536.70 rows=294 width=60) (actual time=0.156..0.376 rows=273 loops=1)
                     Recheck Cond: (term_vector @@ '''intl'':*'::tsquery)
                     Heap Blocks: exact=162
                     ->  Bitmap Index Scan on searches_next_term_vector_idx  (cost=0.00..26.20 rows=294 width=0) (actual time=0.139..0.139 rows=274 loops=1)
                           Index Cond: (term_vector @@ '''intl'':*'::tsquery)
 Planning Time: 0.312 ms
 Execution Time: 6.273 ms

Rails Integration

That’s all for the pure SQL side of things. It’s worth briefly covering how to integrate all of this into a web framework. Since Pirep is written in Rails, I cover that here, but the concepts are generally the same for any MVC web framework in terms of configuring models/controllers and reindexing records. Below are snippets of code pulled from Pirep. Some necessary plumbing code is omitted for brevity. Full code listings are linked to at the bottom.

Indexing

To represent the search data, a Search model is created with a standard migration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class AddSearch < ActiveRecord::Migration[7.0]
  def change
    create_table :searches, id: :uuid do |table| # rubocop:disable Rails/CreateTableWithTimestamps
      table.references :searchable, null: false, polymorphic: true, type: :uuid
      table.tsvector :term_vector, null: false
      table.string :term, null: false
      table.point :coordinates

      # Create a gin index for search performance and an index for upsert statements when reindexing individual records
      table.index :term_vector, using: :gin
      table.index [:searchable_id, :searchable_type, :term], unique: true
    end
  end
end

The Search model is a polymorphic relationship to any other record type that is to be made searchable. Even though reindexing is super fast, with enough data it will inevitably begin to slow down. Since it would be a bad idea to be running live search queries against a search table that is also actively being indexed, this process uses a temporary searches table that is then swapped out with the live table as such:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
def self.reindex!
  search_records = []

  # Collect indexing statements from all searchable models
  SEARCH_MODELS.each do |model|
    search_records << model.search_index
  end

  statements = [
    # Drop and create a new temporary search table by copying the structure of the existing one
    "DROP TABLE IF EXISTS #{TABLE_NEXT}",
    "CREATE TABLE #{TABLE_NEXT} (LIKE #{TABLE_CURRENT} INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES)",

    # Insert the search records for all models (note that `UNION ALL` won't check for duplicates here)
    "INSERT INTO #{TABLE_NEXT} (searchable_id, searchable_type, term_vector, term, coordinates) #{search_records.join("\nUNION ALL\n")}",

    # Replace the current searches table with the new one
    "ALTER TABLE #{TABLE_CURRENT} RENAME TO #{TABLE_LAST}",
    "ALTER TABLE #{TABLE_NEXT} RENAME TO #{TABLE_CURRENT}",
  ]

  transaction do
    statements.each do |statement|
      connection.execute(statement)
    end
  end

  # We don't need the old table anymore
  connection.execute("DROP TABLE IF EXISTS #{TABLE_LAST}")
end

The method above collects indexing SQL from each model that is made searchable. This is defined in a Searchable concern included on the relevant models:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
module Searchable
  extend ActiveSupport::Concern

  module ClassMethods
    def searchable(term)
      self.search_terms ||= []
      self.search_terms << term
    end

    def search_index
      return self.search_terms.map do |term|
        <<~SQL.squish
          SELECT
            id::uuid,
            '#{name}',
            #{term_to_tsvector(term)},
            '#{term[:column]}',
            #{self == Airport ? 'coordinates' : 'NULL::point'}
          FROM #{table_name}
          WHERE #{term[:column]} IS NOT NULL AND #{term[:column]} != ''
        SQL
      end
    end

    def term_to_tsvector(term)
      # Use "simple" language here to avoid mangaling names since these are all proper nouns and specific terms
      term_vector = "to_tsvector('simple', #{term[:column]})"

      if term[:weight]
        # Allow for conditional weighting
        if term[:weight].is_a? Array
          term_vector = "CASE WHEN #{term[:weight][0]} THEN setweight(#{term_vector}, '#{term[:weight][1]}') ELSE setweight(#{term_vector}, '#{term[:weight][2]}') END"
        else
          term_vector = "setweight(#{term_vector}, '#{term[:weight]}')"
        end
      end

      return term_vector
    end
  end
end

This allows searchable models to define their search terms by simply calling the searchable method:

1
2
3
4
# Rank airport codes above names to prioritize searching by airport code
# Also rank public airports over private airports
searchable({column: :code, weight: ['facility_use = \'PU\'', :A, :B]})
searchable({column: :name, weight: ['facility_use = \'PU\'', :C, :D]})

The above is having airports indexed by both their unique codes and also names with codes prioritized over names and public airports prioritized over private airports in the results.

This can be changed to any column for the model. For example, a User model could easily index by email address with searchable({column: :email}).

While it’s possible to reindex the entire database whenever something changes (it’s fast enough at this scale), it’s much more efficient to only reindes the affected records. The Searchable model has a function to do this too:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
after_create :search_reindex!
after_save :search_reindex!, if: :should_reindex?

def should_reindex?
  return search_terms.any? do |term|
    send("saved_change_to_#{term[:column]}?")
  end
end

def search_reindex!
  transaction do
    self.class.search_terms.map do |term|
      statement = <<~SQL.squish
        INSERT INTO searches (
          searchable_id, searchable_type, term_vector, term, coordinates
        )
        SELECT
          id :: uuid,
          '#{self.class.name}',
          #{self.class.term_to_tsvector(term)},
          '#{term[:column]}',
          #{instance_of?(Airport) ? 'coordinates' : 'NULL::point'}
        FROM
          #{self.class.table_name}
        WHERE
          id = '#{id}'
          AND #{term[:column]} IS NOT NULL AND #{term[:column]} != ''
        ON CONFLICT (searchable_id, searchable_type, term) DO UPDATE SET
          term_vector = excluded.term_vector, coordinates = excluded.coordinates
      SQL

      self.class.connection.execute(statement)
    end
  end
end

Because all of the indexing is done directly in the database it’s nearly instantaneous to index tens of thousands of records and incurs none of the usual Rails’ overheads.

Searching

Then to query the data the Search model has a query method which handles a few tasks for us. First, it normalizes queries by always searching in lowercase since search is otherwise case sensitive. It also truncates absurdly long queries that may make Postgres choke and removes bad characters that are invalid search syntax. The caller can also request a wildcard search for allowing partial word matches.

For the results, the query method will consider which models to query. With the exception of a global search, in most cases you want to only search one type of record. In this case, we can return those models directly instead of Search records. However, in the case of mixed model results the Search records are returned to keep an ActiveRecord_Relation object instead of an array. It is the caller’s responsibility to use the Search records as needed to get the relevant associated records.

In the code below, custom rank values are used since I wanted airport searches by code to always significantly outweigh searches by name.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
def self.query(query, models=nil, coordinates=nil, wildcard: false)
  # Normalize casing, escape special characters that will cause syntax errors in the query, and truncate queries that are ridiculously long
  query = query.downcase.gsub("'", "''").truncate(100)

  [':', '(', ')', '<', '>'].each do |character|
    query = query.gsub(character, "\\#{character}")
  end

  # Add a suffix wildcard to the query if requested to allow for partial matches on words
  query = query.split.map {|term| wildcard ? "#{term}:*" : term}.join(' & ')

  # Only for airports: Rank the results by proximity to the coordinates if given any. This uses the `<@>` operator to calculate the distance
  # from the airport's coordinates to the given coordinates with Postgres' earthdistance extension. This assumes the Earth is a perfect sphere
  # which is close enough for our purposes here. This distance is then multiplied by the result's rank such that further away airports have a
  # higher rank and thus show lower in the results.
  #
  # Likewise, when doing the ranking we want to prioritize results for airport codes over airport nodes. The weights are set such that the
  # A and B weights will have higher ranking nearly always.
  coordinates_weight = (coordinates ? "* (point(#{coordinates[:longitude]}, #{coordinates[:latitude]}) <@> #{table_name}.coordinates)" : '')
  rank_column = "ts_rank('{1, .9, .1, 0}', term_vector, '#{query}') #{coordinates_weight} AS rank"

  # If we're given multiple models to search return search records directly. If we're only given one particular model then we can return that model's records
  # This allows to return an ActiveRecord Relation object if needed for further querying or by passing an array with multiple models for display in a mixed
  # global search results page or simply as a way to get the underlying search records for a given search term.
  search_query = if models.is_a? Array
                   select("#{table_name}.*", rank_column).where(searchable_type: models.map(&:name))
                 else
                   models.select("#{models.table_name}.*", rank_column).joins("INNER JOIN #{table_name} ON #{table_name}.searchable_id = #{models.table_name}.id")
                 end

  return search_query.where(sanitize_sql_for_conditions(["term_vector @@ to_tsquery('simple', ?)", query])).order(:rank)
end

When it’s all said and done, the final interface for using all of this looks as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# Making a model searchable:
class Airport < ApplicationRecord
  include Searchable
  searchable({column: :code, weight: ['facility_use = \'PU\'', :A, :B]})
end

# Performing a query:
class AirportsController < ApplicationController
  def search
    coordinates = (params['latitude'] && params['longitude'] ? {latitude: params['latitude'].to_f, longitude: params['longitude'].to_f} : nil)

    results = Search.query(params[:query], Airport, coordinates, wildcard: true).limit(10).uniq
    render json: results.map {|airport| {code: airport.code, label: airport.name}}
  end
end

# Reindexing all records:
Search.reindex!

# Reindexing a single record:
Airport.first.search_reindex!

Full code listings are available at:

In conclusion, Postgres’ full text search has been an extremely performant and flexible tool that let me push complex logic directly into the database that I did not originally think was possible. Moreover, for a small project like mine, being able to have fairly sophisticated search functionality built inside of an existing piece of my stack without needing to use (and pay for) another service is hugely beneficial as well. It’s certainly not the be-all-end-all of search functionality, but it will more than sufficiently handle a large number of search use cases nearly out of the box by only writing a few SQL queries. What else can you ask for?

Coexisting Rails Import Maps with Yarn

Wed, 15 Mar 2023 00:00:00 -0700

I dislike Webpack. I even more strongly dislike Webpack when integrated with Rails. My prior experience with it has always been one of those tools that when it works silently in the background doing what you expect it’s fine, but when it doesn’t do what you expect, well, there goes the rest of your day debugging it and endlessly tinkering with obscure configuration files. Then again, JavaScript in general has never been my forte, but all the more reason for a desire to not have JavaScript tooling continually get in the way of focusing on my actual application.

Regardless, when I started work on my recent project, Pirep, circa two years ago I read about a new gem installed by default with Rails 7 applications: importmap-rails. It described itself as such in its README:

You can build modern JavaScript applications using JavaScript libraries made for ES modules (ESM) without the need for transpiling or bundling. This frees you from needing Webpack, Yarn, npm, or any other part of the JavaScript toolchain. All you need is the asset pipeline that’s already included in Rails.

All of this was absolute music to my ears. I instantly adopted it into my application and forwent any type of JavaScript build system. And it worked wonderfully. There was one rough edge, however: third party dependency management.

Third Party Dependencies with importmap-rails

By default, the documented method for adding a third-party JavaScript dependency to your Rails application with importmap-rails is to use one of two options:

Serving the JavaScript modules from a third-party CDN
Downloading a local copy of the JavaScript modules to be included in your repository and served through your application

In my opinion, serving anything necessary for the functioning of your website through a third-party CDN that you have no control over is simply asking for trouble so I don’t bother to consider that here.

For option two there’s a significant drawback: The packages are downloaded from JSPM and for pure JavaScript libraries this works great. But for any packages that also have associated CSS or images with them, well, you’re on your own to get those resources into your application.

Maybe I’m missing something, but this trait alone significantly reduces the usability of this entire gem if I have to source the non-JS assets of a package myself through other means. The core concept of using import maps is solid, but the method of dependency management completely misses the mark here. Looking backwards though, there’s already a decent tool that does handle this: Yarn. I’m not in love with Yarn by any means, but at least it downloads all of the assets needed for the functioning of a particular JavaScript package.

This got me thinking, what if I could use Yarn to download my dependencies and then use the import maps gem to get those assets into the asset pipeline for Rails to use. Well, good news because doing that is the whole point of this post.

Using Yarn with importmap-rails

The downloading functionality of importmap-rails simply downloads the files for a specified JS package to your vendor/assets directory. From there, your importmap.rb and manifest.js files will read the files in that directory for inclusion into your Rails application.

So here’s the trick: it doesn’t matter what places the files in that directory. It could be the importmap-rails download functionality or it could be another dependency management tool like Yarn.

Yarn, of course, will put files into node_modules so if we want to download our dependencies with Yarn and then have import maps read them without manually copying anything or making a mess of our config files we can just create symlinks from vendor/assets to node_modules. Yup, that’s basically the whole point to this post: a fairly dumb, albeit effective, solution to an annoying problem.

Example Package

Let’s walk through then how this would work with a JavaScript package we want to include in our Rails app. For Pirep, I heavily used Mapbox-gl so I’ll use that as an example here.

To start, we add it to our package.json file as you would with any other JS package when using Yarn:

1
2
3
4
5
{
  "dependencies": {
    "mapbox-gl": "^2.10.0"
  }
}

Then running yarn install will fetch the files and put them under node_modules.

Since Mapbox-gl has a compiled JS file we only need to symlink it into vendor/assets. But this approach can also work with multiple JS files using import statements. My vendor/assets/javascripts directory looks as such:

1
2
3
4
$ ls -la vendor/assets/javascripts
drwxr-xr-x kira kira 4.0 KB Sun Feb 26 23:25:57 2023 .
drwxr-xr-x kira kira 4.0 KB Sun Feb 26 23:25:57 2023 ..
lrwxrwxrwx kira kira  49 B  Sun Feb 26 23:25:57 2023 mapbox-gl.js > ../../../node_modules/mapbox-gl/dist/mapbox-gl.js

And similarly for Mapbox’s CSS:

1
2
3
4
$ ls -la vendor/assets/stylesheets
drwxr-xr-x kira kira 4.0 KB Sun Feb 26 23:25:57 2023 .
drwxr-xr-x kira kira 4.0 KB Sun Feb 26 23:25:57 2023 ..
lrwxrwxrwx kira kira  50 B  Sun Feb 26 23:25:57 2023 mapbox-gl.css > ../../../node_modules/mapbox-gl/dist/mapbox-gl.css

Then we can add the package to our config/importmap.rb file to tell it that references to mapbox-gl should be resolved to mapbox-gl.js:

1
2
pin 'application', preload: true
pin 'mapbox-gl', to: 'mapbox-gl.js'

This works because vendor/assets/javascripts is a default search path for Rails’ asset pipeline.

Finally, in the app/assets/config/manifest.js file we tell it to include all JS files under vendor/assets/javascripts in the import map that is sent to the browser:

1
2
//= link_tree ../javascripts .js
//= link_tree ../../../vendor/assets/javascripts .js

This assumes you are using app/assets for your JavaScript files. This is a relative path so adjust it as needed if you have a different location such as app/javascript.

For the CSS, the good ‘ole Sprockets pipeline continues to work well here. In my app/assets/stylesheets/application.scss file I have the following import:

1
@import 'mapbox-gl';

This pulls in Mapbox’s CSS file from the vendor/assets/stylesheets directory that we symlinked before. vendor/assets/stylesheets is a default search path for the asset pipeline so there’s no additional configuration needed.

And that’s it! From here we can use Mapbox-gl in our own JS files with a simple import 'mapbox-gl'; statement. When the browser loads the page it will have an import map defined:

1
2
3
4
5
6
<script type="importmap" data-turbo-track="reload" nonce="P6km21BKVZz6fWe52Z0eae9iEd1Du2jBG+UW5UVKdQ4=">{
  "imports": {
    "application": "https://cdn.pirep.io/assets/application-c6ab36beca07f0adacc25acb300bd176b60316e3c3b436d3ef30f07818b9a4e6.js",
    "mapbox-gl": "https://cdn.pirep.io/assets/mapbox-gl-945cb90660c81cfd8dc80d59a1ae0b69e43748888e5e63bcf16643a05a24315f.js",
  }
}</script>

This will tell the browser which JS modules to load which allows us to natively use import statements without the need for fumbling around with any build systems like Webpack.

For anyone worried about browser compatibility, import maps are currently widely supported in recent browsers. At the time of this writing the only major browser to not have support is Safari (althought support exists in the technology preview version so it’s coming shortly there too). The importmap-rails gem has a built-in polyfill for unsupported browsers though so there’s little need to worry about lack of support here.

Where it won’t work

It’s not all rainbows and sunshine, unfortunately. There are some JavaScript packages that don’t play nicely with import maps just yet.

For example, in Pirep I use Sentry for error reporting. They have a JavaScript library for collecting frontend errors. Unlike Mapbox-gl, however, its NPM package does not have a single built JS file to symlink into the vendor directory. That’s not necessarily a problem though since the whole point of an import map is that it can pull in multiple JavaScript modules directly from the browser. The issue I found with the Sentry library in particular is that some of those import paths are not compatible with loading from a browser. I outlined all of my different attempts at making it work in an issue I opened asking for import map support.

In the end, I ended up pulling the built Sentry file from their CDN and copied it directly into my vendor/assets/javascripts directory then included a separate, standalone <script> tag for it in the <head> of my templates. This was disappointing, but this was also the sole library that I had to do this with so it was only a mild annoyance than a common occurrence.

The other area where import maps won’t do much for you is if you’re using any form of JavaScript that won’t run in the browser. For example, if you make use of TypeScript. The idea is that you’re running JavaScript in the browser as it is written without any intermediary. If you have any compilation step this won’t work for you.

Overall, in my experience so far using importmap-rails has all been a huge simplification of my asset management. Having a Rails application with a clean and understandable asset pipeline takes me back to the pure Sprockets days which I’ve honestly sorely missed. That’s not to say this is ready for everyone to use or for complex legacy applications, but there’s finally a light leading us out of the Webpack darkness that we’ve all been stuck in for the past decade.

Generating map tiles for FAA sectional charts with GDAL

Mon, 13 Mar 2023 00:00:00 -0700

Recently I launched the initial version of Pirep, a collaborative website for pilots to collect & share their local knowledge about airports such as transient parking location, crew car availability, nearby attractions/restaurants, camping information, etc. The central component of Pirep being the map page where airports are charted and filterable based on what amenities exist at them.

While it’s easy to get a map with satellite imagery, naturally one would also want the FAA VFR sectional charts as a layer on the map as well. This turned out to be fairly easy to get a proof of concept for, but much more tedious to get to a production-ready state.

If you’re looking for the final product, skip to the bottom.

As every pilot knows, the FAA publishes VFR sectional charts which divide the continental US into 37 rectangles of roughly equal size. This division being a left over from the era when pilots used paper charts for navigation. It wasn’t exactly practical to fumble through a map the size of the entire country in a small cockpit and also not cost effective to buy a new paper chart of the whole country every 56 days when most private pilots are only flying in one section of the country on a regular basis.

As such, the FAA publishes digital map files now for each of these divisions. Thankfully in a GeoTIFF format too which makes them fairly easy to work with using industry standard tools. All of the files are located on the FAA’s Digital Products website.

With that, the first step to making our map layer would be to convert one of these charts to map tiles and put it on a web-viewable map.

Enter GDAL

This entire process boils down to using GDAL utilities in the proper manner. GDAL being a open source library for manipulating geospatial data. GDAL is an incredible resource and none of this would be possible without their software.

It’s mostly straightforward to convert one of the FAA’s GeoTIFF files into map tiles with GDAL’s Python script, gdal2tiles.py. For example, if you download the Seattle sectional chart from the FAA’s website, we can generate map tiles as such (for these examples I’m using the Seattle chart but the process is the same for any chart):

1
2
3
unzip Seattle.zip
gdal_translate -of vrt -expand rgba "Seattle SEC.tif" Seattle.vrt
gdal2tiles.py --zoom "0-11" --processes=`grep -c "^processor" /proc/cpuinfo` --webviewer=none Seattle.vrt tiles

Note: the --processes option is used here with the value set to the number of CPU cores on the system to speed up generation. This can be removed/modified as desired.

Second note: Everything here is written with GDAL version 3.6 or later. Some features used were added in this version. Using an older version will likely result in malformed map tiles.

In order to use gdal2tiles.py we first need to convert the GeoTIFF file into a VRT file which is where gdal_translate comes into play. From there, we can generate map tiles and display them with a web mapviewer. In this case, I opted for MapLibre, but other maps should work as well.

Looks pretty good, huh?

I’d say we have a decent proof of concept. We can take a GeoTIFF file from the FAA, generate map tiles, and display them in a browser. It should be straightforward to add the rest of the sectional charts to get our sea-to-shining-sea map, right? Well, things start to get more complicated from here unfortunately.

Displaying Multiple Charts

My first attempt at displaying multiple charts was to simply run the same GDAL commands as above for each chart with the same output directory. After all, map tiles are just small images with unique paths. It should be fine to build up a directory of these images through multiple passes, right? To a point that works, but the main problem comes with the boundaries between charts. Since there will be some overlap between the tiles neighboring charts would overwrite each other creating gaps in the final map. That’s no good.

Instead, most web map viewers support the concept of multiple layers. So we could generate map tiles for each chart in its own directory, add each one as a layer on the map, and call it a day. This works to an extent but has two significant problems:

Performance. Creating a map layer for each chart was my second attempt at solving this and while it worked decently from a visual perspective, my experience was the map viewer I was using, Mapbox, started running into significant performance issues when you put 40+ map tile layers on the page. This worked, okay-ish, but the initial page load time could be upwards of 10 seconds which was unacceptably slow for my purposes.
The bigger issue, however, is the chart legends in the GeoTIFF files. If we ignore these they’ll have tiles generated for them as well and then we end up with something that looks like this:

That clearly looks awful, but these are layers after all, so why not layer them in a particular order such that the legends are overlapped by neighboring charts in the right order? Again, this kind of works (seeing the trend with this yet?), but the problem is that the sectional charts are not consistent with their legends and axes. You’d need to load the charts in a very particular order to get this to look right and that’s also not accounting for the odd-shaped charts too which aren’t perfect rectangles. Maybe you could get this to work, but combined with the performance issues meant that I had to look for another solution.

Chart Cropping

The solution to the problems above is two-fold:

Crop the legends and axes out of the GeoTIFF files so we’re left with just the actual chart portion.
Combine all of these cropped charts into one big chart and then use GDAL to generate tiles from it since it’s thankfully smart enough to handle stitching together multiple charts seamlessly where they overlap.

Cropping the charts is easier said than done, however. The charts themselves are not of standard sizes and vary with latitude. GDAL can do this cropping, but we need to give it a shapefile to crop against. And since there’s no way to programmatically do this given the variability in each chart (and why this blog post will save you a bunch of time if you’re looking to replicate this yourself), we have to manually create a shapefile for every. single. chart. Long story short, I fired up QGIS, imported the GeoTIFF charts, and manually created shapefiles for each one representing the actual chart portion. With these (linked to at the bottom), we can then tell GDAL to crop each chart accordingly before combining them into a single chart for further processing.

First though, to demonstrate let’s crop one chart and generate tiles from it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
unzip -o "Seattle.zip"

gdalwarp \
  -t_srs EPSG:3857 \
  -co TILED=YES \
  -dstalpha \
  -of GTiff \
  -cutline "shapefiles/Seattle.shp" \
  -crop_to_cutline \
  -wo NUM_THREADS=`grep -c "^processor" /proc/cpuinfo` \
  -multi \
  -overwrite \
  "Seattle SEC.tif" \
  "Seattle cropped.tif"

gdal_translate -of vrt -expand rgba "Seattle cropped.tif" "Seattle.vrt"

gdal2tiles.py \
  --zoom "0-11" \
  --processes=`grep -c "^processor" /proc/cpuinfo` \
  --webviewer=none \
  "Seattle.vrt" \
  tiles

Here, we’re now employing gdalwarp do some preprocessing on the chart before sending it to gdal_translate. We’re also reprojecting the chart to EPSG:3857 with the -t_srs option. This is necessary because without this when the tiles are generated they won’t be projected on the globe correctly and result in an odd looking map like this:

Back to the cropping, the star of this show is the -cutline and -crop_to_cutline options where we give gdalwarp our shapefile for the chart in order to get rid of the legend and axes. Everything that follows is the same as before and then we’re left with the following:

Combining Charts

Awesome, so now we’re ready to combine all of the charts into a single TIFF file and generate charts from it. This took some trial and error on my part to handle the color and alpha channels properly, but the commands below handle all of that now.

Note that generating map tiles for all sectional charts of the continental US and Alaska would take anywhere from a few minutes to a few hours depending on your computer’s resources and uses about 10gb of disk space when it’s all said and done. Because of that only four sectional charts are used here for demonstration purposes, but the process is the same for all charts.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
CHARTS=("Seattle" "Klamath Falls" "Great Falls" "Salt Lake City")

rm all_charts.vrt || true
rm -rf webviewer/tiles || true

for CHART in "${CHARTS[@]}"; do
  unzip -o "$CHART.zip"

  gdalwarp \
    -t_srs EPSG:3857 \
    -co TILED=YES \
    -dstalpha \
    -of GTiff \
    -cutline "shapefiles/$CHART.shp" \
    -crop_to_cutline \
    -wo NUM_THREADS=`grep -c ^processor /proc/cpuinfo` \
    -multi \
    -overwrite \
    "$CHART SEC.tif" \
    "$CHART cropped.tif"

  gdal_translate -of vrt -expand rgba "$CHART cropped.tif" "$CHART.vrt"
done

gdalbuildvrt all_charts.vrt *.vrt

gdal2tiles.py \
  --zoom "0-11" \
  --processes=`grep -c ^processor /proc/cpuinfo` \
  --webviewer=none \
  all_charts.vrt \
  tiles

And with that, we have a nicely tiled, continuous map for multiple sectional charts:

It’s not perfect; if you look closely at the boundary between two charts it’s visible where the boundary line is. However, the same artifacts can be seen on other web-viewable sectional chart websites like SkyVector so I’m fairly confident they’re using the same type of process as is done here. And frankly, this is good enough for virtually all purposes with these charts.

Maybe in the future the FAA will publish one GeoTIFF with everything already combined. That would certainly simplify this process considerably.

Tile Optimization

There are some further optimizations that we can make though. With the generation commands above, we’re left with a total directory size of just under 1gb for the four sectional charts. Each tile is on average around 100kb.

A new feature with GDAL v3.6 is the ability to use WEBP images instead of PNGs for tiles. This results in a significant reduction in image size. That’s nice for the merit of using less disk space, but the real benefit to this is that a smaller tile size means less network transfer time and a much quicker loading map. Plus, since these chart images are not especially detailed images we can turn up the compression on them without significantly affecting image quality. By doing this that 1gb total size for four charts can be reduced by a whopping 90% to right around 100mb!

Another notable option to enable is --exclude. This will tell GDAL to skip any tiles that would otherwise be empty. Even an empty image will still use a small amount of disk space and when you add all of these empty tiles up that can add up to be a significant amount. This isn’t a big deal when dealing with a continuous chart like the continental US since there would be minimal empty map tiles, but if you throw the charts for Alaska, the Caribbean, and Hawaii into the mix there’s a ton of empty space between those charts which GDAL would otherwise generate blank map tiles for.

Using both of these options, the gdal2tiles.py command reads as follows:

1
2
3
4
5
6
7
8
9
gdal2tiles.py \
  --zoom "0-11" \
  --processes=`grep -c ^processor /proc/cpuinfo` \
  --webviewer=none \
  --exclude \
  --tiledriver=WEBP \
  --webp-quality=50 \
  all_charts.vrt \
  tiles

Other Chart Types

It’s worth mentioning that the same steps for generating map tiles will apply to other chart types that the FAA publishes in a GeoTIFF format. For example, in Pirep the map has the terminal area charts available on it as well. These are generated with the same process: Crop the charts with a shapefile, combine into a single VRT file, generate map tiles. The difference being that only zoom levels 10 and 11 are generated for these charts since they only show on the map when sufficiently zoomed in. Given the physical distance between these charts, this is where the --exclude option from above makes a significant difference by skipping all of the tiles that would be empty without it.

Demo Code

To sum it all up, I put together a self-contained demo you can try locally yourself.

This archive contains a Bash script for generating map tiles, the shapefiles for cropping the charts, and a small HTML page to view the map in a browser. To use it:

Download and extract the archive.
Download the Seattle, Klamath Falls, Great Falls, Salt Lake City sectional charts from the FAA’s digital products page. Save these in the same directory as the extracted archive. The map tiles generation script will extract these for you.
Run ./generate_map_tiles.sh
Run cd webviewer && ./run_server.sh (this starts a small Python webserver)
Open localhost:8000 in your browser

Additional Resources

Everything above is a minimal demo for generating map tiles. To fully production-ize this additional configuration and logic is needed to handle all of the charts and their insets, plus all of the shapefiles for each chart. These are linked to below as well as the relevant code in Pirep for generating the map tiles.

ephemeral.cx

Losing Access To The Cascades

Scope

How did we get here?

Budgets & Appropriations

Region 6 & Mt. Baker-Snoqualmie National Forest

Wildfires

Recreation Passes

Other Funds

Roads and Trails Fund

Legacy Roads and Trails

Bipartisan Infrastructure Law

Inflation Reduction Act

Great American Outdoors Act

Budget Summary

Potential Mitigations

1. Extend the GAOA Legacy Restoration Fund

2. Use the Land and Water Conservation Fund for maintenance

3. Increase enforcement of recreation passes

4. Raise the cost of recreation passes

5. Strengthen the Roads and Trails Fund Act

6. Make use of the Highway Trust Fund

7. Allow more concessionaires

8. Increase sustainable logging

Mt. Pilchuck case study

Conclusions

Sources

Hey Google, what happened to all the fun?

Debugging Ruby, The Hard Way

But why would you want to do this?

GDB Basics

Getting a Ruby Backtrace

What about other threads?

Into the VM

Tracking Down a Line Number

Getting a File Name

Five of the best fly-in camping airports in Washington

1. 09S - Sullivan Lake State

2. 6S9 - Stehekin State

3. KORS - Eastsound / Orcas Island

4. 4S6 - Tieton State

5. 3W7 - Electric City / Grand Coulee Dam

Bonus Airport: 3S7 - Nehalem Bay State / Manzanita

Shameless Plug

The feasibility of electrifying Mt. Baker

Why is this necessary?

How much CO2 does Baker emit?

Electrification feasibility

Is it worth it?

What about hydrogen?

Conclusions

Sources

Five of the most accessible Idaho backcountry airports

1. 1U1 - Moose Creek

2. 3U2 - Johnson Creek

3. S81 - Indian Creek

4. U60 - Big Creek

5. 0U1 - Warm Springs

Bonus Airport: 8U2 - Schafer Meadows

Shameless Plug

Sources

The case for reestablishing winter access to Mt. Pilchuck

Mt. Pilchuck History

Current Access

Why create more winter recreation access?

Why is Pilchuck the right place for this?

Economics

Conclusions

Sources

Geographically ranked Postgres full text search

Searches Schema

Indexing

Searching

Ranking Results

Ranking by Distance

Rails Integration

Indexing

Searching

Coexisting Rails Import Maps with Yarn

Third Party Dependencies with importmap-rails

How much CO₂ does Baker emit?