hckrnws
I'm in Fort Collins and have had city fiber for a couple years now and it's a game changer. We got it during earlyish covid and having 4 of us on video calls was bringing Comcast gig service to it's knees, I imagine because of the limited outbound bandwidth. The city fiber service is gigabit symmetric, no caps, $70/mo, or 10gig for $200/mo. They also are introducing a 2.5gbps, I think that's $100.
I'm working on upgrading my internal network to be able to take advantage of upgrading to 10gig.
And one of the best parts is that when I had a weird networking issue, I was on the phone with a legit networking person in 15 minutes, rather than stuck with someone who couldn't do anything more than read from a script.
The issue ended up being a Comcast routing issue, which I told them from the top, but figured they'd like a heads-up about it. Me and one of my coworkers on the city fiber couldn't reach the office on Comcast, but 2 other coworkers also on city fiber could. I eventually got ahold of a Comcast Business network engineer later that day and they had it resolved the next day.
Why do asymmetric plans exist in the first place anyway? I assume connections between autonomous systems are symmetric full-duplex, so doesn't these mean ISPs have the same amount of uplink bandwidth to divide between customers as downlink?
A few points:
a) Most non-fiber connection technologies have limited bandwidth to split between upload and download. If you are offering connections over those, offering a symmetric speed means you are offering a lower max speed (years and years ago, in the single-MBit days, there was a regional German ISP that let you switch your ratio - was very popular with developers and designers at the time, who could reset their connection to be high-upload before sending large files)
b) Only a subset of users cares about high upload speeds (To a degree it's a self-fulfilling prophecy: people are so used to it that most don't even know why they'd want good upstream because they never had the opportunity to do something that would benefit)
c) Some that really want them can be made to pay a lot extra (e.g. for "business class" service, especially if combined with things like static IPs for hosting)
d) some high-bandwidth uses are more likely to cause work for the ISP: people hosting servers (complaints about content etc, attracting attacks), file sharing (rightsholder complaints). Similarly, a customer machine turning into a DDoS traffic source has worse impact if it has high bandwidth
Before cables modems everything was symmetric. Cable modems were bandwidth constrained and they decided to offer more download then upload. One could also argue that the phone companies as network companies and wanted to make connections and cable companies just wanted to provide content. Offering symmetric bandwidth could be seen as competition by cable companies becuase anyone can publish. Finally the phone companies started to offer asymmetric bandwidth when DSL was introduced. One could subscribe to asymmetric DSL or symmetric DSL.
> Before cables modems everything was symmetric.
Late 1990s V.90 modems were asymmetric, higher download speeds caused upload speed to deteriorate, and often up traffic would be shunted into a more limited analog channel to keep it from interfering with the down speed. https://en.wikipedia.org/wiki/Modem#Standardized_56k_(V.90/V...
I remember something in the earlier 14400/28800/USRobotics era being asymmetrical too, but can't find a reference.
Where I've lived, ADSL also predates cable modem, and the A there is for Asymmetric.
One component of it can be the exchange and transit deals that the ISPs make. "Peering" is generally made under the terms of "sending as much as you receive", so in that way it can be beneficial to limit your outbound, if you are trying to slip into available inbound bandwidth to pair with some business-class outbound-heavy.
But, there are also lots of other business and technology reasons as others go into here.
Because GPON is asymmetric and that is the cheapest way of deploying fiber optic for residential use.
If you want symmetric, then you need a lot of "active" equipment and that drives the costs up.
GPON can run symmetric just fine, but the customer-side equipment does get more expensive with higher speeds, true.
GPON is limited to 2.4 gbps down, 1.2 gbps up. It's asymmetric.
GPON has a bunch of modes, some of which are symmetric. Notably a 1.2 Gbit/s down/up setup for gigabit service. Many modes are choosen to be asymmetric, yes, but the technology is not intrinsically one or the other. Notably, just like non-multiple-access fiber connections increasing upstream speed doesn't negatively impact downstream speed (as it does for copper-based connections).
Why use 1.2/1.2 when you can use 2.4/1.2 ? It makes no sense whatsoever. Reducing download so you can say your connection is symmetric?
In any case GPON is already in its way out, new deployments use XGS-PON which is 10/10
2.4/2.4 is also a standardized GPON speed... Just not very common.
But either way it's irrelevant to the point that the technology can very well support symmetric speeds and offering symmetric doesn't "need a lot of "active" equipment and that drives the costs up." as the commenter I originally replied to claimed. Also evidenced by XGPON having the exact same amount of active equipment.
For what is worth, I haven't had much contact with technology in the last couple of years and things might have changed. My comment was based on my last experience where GPON by design was asymmetric and if you wanted symmetric speeds you would need an MC on both ends of the fiber, hence de "active equipment" need. And then from the MC to a router or whatever you had there.
On the other hand, more than 300Mbps upload is quite ok even with heavy video conferencing and what not.
Because technologies like GPON or HFC were designed with asymmetry in place because most users download much, much more than they upload. In the case of HFC the asymmetry can be particularly egregious.
I'm working on upgrading my internal network to be able to take advantage of upgrading to 10gig.
I'm curious about the best way to do that. What are the upgrades you're thinking about doing?
Sure, I'm happy to go into it.
Prior to 2 years ago I had been running cat5e shielded cable around the house (security cameras wanted shielded). cat5e for shorter runs apparently can do 10gig. But once I heard I'd be able to get 10gig, I started doing runs as cat6a. Significantly more expensive, but YOLO.
I got an Aruba 2500 PoE switch for ~$100, it has two 10gig SFP ports.
I have an old Dell R720 coming that I'm going to put a $75 dual 10gig card in. My plan is to hook that up to the fiber ONT, and run Proxmox or Ganeti on it, then run a VM for my firewall. Then hook that to the switch at 10gig.
Currently I don't have any interior devices that will do 10gig, so there's not much day-to-day use I'll see from it currently. When I get there, I'll probably be looking at adding a Mikrotik 10gig switch because they're relatively affordable, though maybe I'll be able to find some old enterprise gear as well. So far I haven't really seen any. The Netgear 10gig switch we have at the office seems to work fine and was ~$800 new, but is EOL, so maybe something like that will be cheap. It's only fallen over once. :-)
So the big question is why I'd go to 10gig... My primary machines are laptops, my work machine stays on my desk, so I could add one of those $300 USB-C to 10gig adapters, but probably won't. My primary personal machine is a macbook I use 100% wireless, so not really going to get any better there until I replace my Ruckus R610 with something that'll do 2.5gbps, but no plans for that anytime soon.
So the Dell could have 10gig fairly easily. Maybe I'll mirror some OpenStreetMap data or mirror media for work to cut down on our AWS costs... That could be done at 1gbps though as well.
Partly, I just want the geek cred of 10gig, partly I want to support the city fiber network through paying more for a bigger plan. Maybe I'll offer my family offsite backup, but that also probably won't need 10gig.
10gig switches, cat7 cables/fiber cable and 10gig ethernet cards for computers?
Microtik and Xyzel switches, MM fiber and SFP+ modules, and ConnectX-3 NICs from ebay. I did this last year and it was cheap and has been working great.
> 4 of us on video calls was bringing Comcast gig service to it's knees, I imagine because of the limited outbound bandwidth...upgrading to 10gig
Huh, what? Stop imagining and do the math with numbers, maybe Comcast is shitty in general and other metrics.. but still doing here regularly 2 concurrent video calls with sometimes Netflix on top without any issues... on 50 Mbit still! (also asymmetric btw, I may be undemanding, but so a bit wtf on reading that)
(Or Comcast cable internet? There is often more bandwidth sharing involved than they tell, especially upstream, or what does gig service mean?)
Comcast gig is 1gb/s downstream. I don't know what the upstream rate is and they avoid mentioning it, but based on my experience with them I could see 4 video calls being enough to choke it. It's hard to really appreciate the symmetric data rate until you really start using some decent upstream traffic, then it becomes a day and night kind of difference.
To pile on the examples, I had a Comcast connection in SF that was 1gbit down, 5mbit up during covid (based on regular speed tests and what little I could glean from the plans without being the account holder). With 6 people in the house juggling video calls there was definitely some audio only days when schedules clashed.
I still don't get all the downvotes.. if Comcast is selling "gig service" and you only get thst little upstream that is hilarious even in the most extreme asymmetric scenarios...
If you now have symmetric 1 gig is more than enough for any private use, why would one consider 10 gig for home?
> why would one consider 10 gig for home?
For the sheer crazy joy of it!
As if human wasting is not already enough, got it :)
It depends on whether or not they consume more as a result really - think about it, is the number of joules to transmit down a wire different at different speeds? I'd guess probably not
It wasn't that serious... and still need to process 10-100 gigabytes files for work sometimes, and am doing fine with my 50 Mbits... I just cannot imagine one person really.making good constant use of 2.5 Gbit, and in the rare times needed not just invest a little time, so I think it is money wasted in this case. Admitted, my imagination may be too limited and it is a valid tradeoff of time vs money.
But yes, nitpicking, but on the same tech it should be insignificantly more joules also, different techs or hw much more significant though.
In Chicago it was typically 40Mbps but in the last few weeks my scheduled speed tests are running 230Mbps. However, during Covid I had co-workers that lived in communities with a large number of WFH and they were all having trouble with video conf on Comcast so it doesn't surprise me at all.
Upstream from Comcast 1gbps plan was 15mbps when I had it.
Thankfully I’m now in a building with Google webpass that has 1gbps symmetric (and no data cap) - much better.
In Greeley (30 min southeast of Foco) max upload with 1gbps is 20mbps.
As others have said, it is the outbound that was really killing it. Sure, we could watch multiple Netflix, but streaming our camera video and audio and it'd get flaky. ISTR I had 250Mbps inbound at the beginning and outbound was ~25Mbps. Upgrading to gigabit got us the fastest outbound they offered at the time, I think it was 40 or 50Mbps. Also, as others have said, Comcast really doesn't like talking about the upstream speed. When I got the upgrade to gigabit, I did their chat support to ask them and even when their support people were directly asked they were reluctant to say. ISTR they kept asking why I wanted to know.
I'm on Spectrum with 1 gig down, 25 Mbps up. It's not difficult to saturate my upstream.
Back when we had 100mbit down with comcast, it couldn’t reliably stream at 3mbit.
> Senate Bill 152 was promoted by the cable industry as a way to prevent wasting taxpayer money on infrastructure projects, like municipal internet.
Yes, I'm sure it was to "not waste taxpayers money" and not to remove competition...
Hell, why cable companies had a say in that in at all?
It's hard to remember but this is the level of tech discourse in 2010. Though I guess it's not any better now.
The saddest part is that he's still in power.
I’d be particularly curious if there were any projects in between that truly were a “waste of tax payers money”. That should tell you something.
Consider that unrelated to all the slavery stuff, Confederacy had constitutionally banned infrastructure improvements by government...
Companies are people and money is speech? And we’re all very free.
Because in a capitalist society, capital owners are in control
Can it be called democracy then?
Do you feel represented ?
I sure don't
I’m not either and i’m a citizen in both the Us and France.
They call it that to placate the masses
[flagged]
In Denmark we now have an infrastructure / network-provider split on fibre, so that one company provides the physical infrastructure and then there is an open marked for network providers to provide services over that fibre.
Physical last-mile infrastructure is owned and operated by utility companies (privately operated non-profits), typically together with electricity or water, but in some areas by specialized fibre utility companies.
This setup requires a mature technology, but it has been very effective in driving consumer costs down.
Japan does this for fiber too. You pay an annual fee for the fiber itself and a separate fee for the ISP to provide the bits. Provides a lot of competition for the ISP-portion and Japan's had gigabit FTTH earlier than most (it helps that Japan's geography is such that most people are in major cities.)
Yes, this is one of best ways to do fiber. The fiber provider digs all the trenches and runs fiber to every building/house in the city and then you as an end-user can contract the internet service from whichever provider you want.
The costs are minimal and in my neck of the woods I pay about $10 for 1Gbps (actual rates being about 950mbps down and about 800mbps up).
In Estonia we are trying to do similar thing with infra/isp split. Coverage is limited but already those places can get 1Gbit internet 3X cheaper than others.
NextLight is truly an incredible service.
It is proof that municipal internet can be awesome and is probably the way most municipalities should go over time.
For those who don't know, Longmont Colorado's Nextlight internet service provides symmetric 1Gb/sec speed for 50 USD per month and they don't spy on your traffic.
It is the most reliable residential internet service I have had to date and is more reliable than the commercial internet at my places of work in Boulder.
I don’t brag on Chattanooga, TN often, but when I do, it’s always about the municipal symmetric gig fiber—CHA/EPB was the first to do it in the US, it’s always been cheap, it’s more reliable than anything else anywhere I’ve been, no data caps or throttling, held off a Comcast lawsuit trying to stop them (so Comcast went to the state to legislate away meaningful competition), and (afaik) there’s no spying on you.
I was lucky enough to build a bunch of the software used consumer-side to enable signing up for and managing fiber internet/tv/phone service, was brought in to establish and run their first internal software team, and built even more cool software to make tech support real-time, improve the lives of CSRs, and more. And they’re now getting into quantum computing. Truly a great group of people trying to take care of their city, and every time I run into anyone, I’m still impressed by what they’re doing.
Moved here 4 years ago and the difference between them and comcast is insane.
On one of the Chattanooga FB groups someone will occasionally ask which ISP they should use, and there will inevitably be 10+ pages of people writing EPB, EPB, EPB!
Haha. That doesn’t surprise me at all. There is simply no comparison. Hope CHA is treating you well beyond just the internet!
It's $70 per month. The $50 rate is for only for customers who joined years ago. If you move to Longmont and sign up as a new customer you'll be paying $70.
The pricing isn't super cheap, but it's very fair and the service quality is good. No data caps, symmetrical speed, good reliability. No IPv6 support though. They also recently started offering 2.5 and 10 gbps for $150/250.
$70 for symmetrical 1Gbps sounds cheap to me.
For comparison ATT does it for $80/m (and they support IPv6). It's not really the price that's particularly special part of these services, it's getting the fiber made available in the area. 5 gbps symmetric is $185 which is quite competitive with those prices as well.
Though I'd pay an extra $20/month to just not have to deal with ATT anyways, all else being equal.
Do they let you plug your own router into their equipment though?
Their FTTN stuff definitely would only work with a router that somehow managed to break third party NAT, even in when running in its bypass mode.
(Out of 7 ISPs at over a dozen sites, this is the only time I have had this issue. Support couldn’t diagnose the issue.)
Plug your own router? Sure, put the BGW320 into passthrough mode and 3rd party NAT works fine. You get the public IP on your device and there is no double-NAT. The broken bit is, regardless where you NAT, you still run into the 8,192 session table limit of the device anyways. The only way around that is via unofficial means (which is part of the $20 to just not deal with ATT. The hardware could damn well do passthrough without that limitation). At least that's something you can workaround to a true fix though as opposed to no IPv6, where your best bet is tunnel somewhere.
In that case, they are significantly better than in the past.
That’s still so bad I think they should lose their right of way agreements for not providing service, but it is slightly better than before.
> The only way around that is via unofficial means
Can you explain what to do exactly if you have a Solarflare and a fiber module?
You grab your own 10G-PON ONT box like the Azores WAG-D20 which lets you set the MAC and ONT values to the same as the ATT one. Works pretty good, only complaint I have is there aren't any "ONT on a stick" (i.e. SFP+ module ONTs) that let you do that last part so it's still a box with a separate power supply adapter.
I live somewhere with good competition between private companies and can get "up to" 10 Gb/s symmetric for $70/month. Realistically it's more like 5-7 Gb/s but I'm not complaining.
So you can get good results with municipal. Or in dense areas private works fine if there's competition.
What is the data cap?
2.6784 petabits is the most you'll be able to pull in a month ;)
there isn't one
+100 Yay, Longmont!
My service has been rock solid: https://photos.tylercipriani.com/2023-05-25_next-light.png
Another longmonster here. Totally agree that NextLight is awesome. However, as someone that runs a nonprofit in town, I wish that business/nonprofit pricing was more reasonable.
a fellow longmonster!
There are dozens of us! Dozens!
Rocking the 2.5G service, but occasionally getting up to 2.7 if they're feeling nice. Do wish that they would get IPv6 rolled out already.
We are legion! Love my NextLight.
What's it like living in the big city? I'm just a little Niwoter over here with no hope of ever getting Nextlight.
I'm in Erie and get Gigabit. It's not symmetrical, but it's the best internet I've ever had.
I just bought a house in Longmont a few weeks ago! You like it?
How is Longmont as a place to live? I'm talking about quality of life issues.
NextLight is a local gem!
It's weird but there used to be a lot of private roads. I hope one day we'll see private internet infrastructure as just as weird.
Private roads aren't weird most people use them every day as that is how most people reach homes or businesses they wish to visit.
You know what I mean.
I had a brief moment of hope that this might mean decent internet would be available to me, before remembering the entire reason I don't have fiber internet in-unit is because my condo HOA is a bunch of boomers who turned down the install. Kickin' it 2007-style with my DSL
This is infuriating and would have led me to start a smear campaign against the leadership ultimately leading to a vote of no confidence.
As a filthy renter, I have very little leverage with the HOA unfortunately.
This is infuriating, and if I were a filthy renter would have led me to start a smear campaign against the leadership ultimately leading to a vote of no confidence.
And this being HN, the odds are high that you or someone you're close with already knows precisely how to leverage the outrage of boomers to your advantage.
Think of it like stealing from the rich to give back symmetric broadband to the rich. You'll become a legend of the HOA; babies and access points will be named in your honor.
My landlord is actually a former ISP contractor, and he told me I should put up flyers for Ting and just get everyone to ask the HOA why we didn't have it... But that sounds like work and my lease is up in March anyway
We have a local quasi-municipal phone/Internet provider and they're about twice as much as 500/500 fiber that just got laid, and I'm getting hard pressed to stay with them even though I like the idea.
Maybe I should call and ask if I can keep my current service at the fiber prices.
Does any state (short of maybe California, I suppose) make it on the front page of HN as much as Colorado?
Well, we're a very polarized red/blue state, a tech hub, and the source of most of the water for the entire southwest.
Colorado has not been a purple state in 8+ years and is going more solid blue by the year. In the 2022 election, there was one thought to be close statewide election. It was a blowout, 55.9% dem, 41.3% GOP. The 2020 presidential election had very similar results.
Aggregate stats are nice when it comes to presidential, but we still have a bunch of reps like Boebert who cause havoc in DC.
Sure, but every US state has such a local split. It’s relative to the proportion and size of dense urban areas, and has been at least since the early aughts. But every “coastal liberal” state (or whatever the “blue state” signifier is) swings sharply conservative as you trek out to less populated areas, with very rare exception. And every “red state” has at least a small island that deviates leftish, even Wyoming.
Clearly you haven't been to Colorado Springs, the second largest city in Colorado
I haven’t, but a few searches seem to reinforce my point. What am I missing from afar?
Think of Denver (700k) vs. Colorado Springs(450k) like SF(800k) vs. Oakland (450k) but unlike Oakland which is __roughly__ an extension of SF politics, CO Springs is deeply conservative to Denver's deep liberalism. CO Springs is home to the Air Force Academy and a number of military installations which underlies its politics. Its a "dense urban area" whose politics happens to be conservative. The skew left or right in Colorado typically comes from voter turn out in other areas of the state, e.g. Arapahoe, Jefferson, Boulder, Douglas and the swing of independent voters which makes up 45% of the electorate.
There are cities which run conservative. Bakersfield comes to mind in California, also roughly the size of CS, at 403k population.
The urban/rural alignment of liberal v. conservative politics is a good general rule of thumb. There are always exceptions.
But even (or especially) in deeply-conservative states, cities tend to be far more liberal. Salt Lake City (Utah), Austin (Texas), and Atlanta (Georgia) are particularly strong examples.
Aggregate stats give a clearer picture of statewide politics though. Boebert wasn't elected by the whole state, only the 3rd district.
And we have two Democratic senators now. That's perhaps the best indicator that Colorado is hardly polarized.
Comment was deleted :(
Colorado used to be very polarized, but that's not so true anymore. It leans pretty strongly blue now:
> Biden won Colorado with over 55% of the vote, and by a victory margin of 13.50%, an 8.6 percentage point improvement on Clinton's victory in the state four years prior, the strongest Democratic performance since Lyndon B. Johnson in 1964, and the first time that it voted for a presidential candidate of either major party by a double-digit percentage since Ronald Reagan in 1984.
> In this election, Colorado weighed in as 9.1% more Democratic than the nation as a whole. The results established Colorado as a Democratic stronghold, rather than the Democratic-leaning battleground state it had been for the past three election cycles.
https://en.wikipedia.org/wiki/2020_United_States_presidentia...
Good lord. Not voting for trump doesn't make us "blue".
It helps that the Colorado Republican party is about as ineffective of a political group as I've ever seen. If they were even moderately competent Boebert's district, Colorado Springs, and to a lesser extent the new district (Caraveo's) would be much redder than they voted the past cycle.
The national political lens is probably the absolutely least informative way to understand Colorado politics. CO politics isn't just a miniature version of the national debate. Our governor is a libertarian in all but name. 96% of the bills in the legislature were bipartisan sponsored last year. 45% of the state is unaffiliated with a party and in 2022, the majority of them voted in the republican primary over the democratic one. The rest of the electorate is split down the middle D and R.
So, I just manually checked via Algolia for total story mentions for all time as of this writing. Keep in mind that this is not front-page mentions necessarily, though it might be possible to crawl the "past" links to get those: <https://news.ycombinator.com/front>.
Maine (22,890) and Iowa (19,353) both beat out Colorado and California. Why the Maine love I've absolutely no idea.
"Washington" appears 22,246 times, though that's massively confounded by appearances of "Washington Post" (18,634). Adding "state" as a qualifier (e.g., "state of..." or "... state") reduces that to 461 instances. "Seattle" alone appears nearly 5,000 times which would be 9th overall (sorry, Florida). "Redmond" gives another 892 occurrences.
Colorado is actually 11th most frequently occurring state name.
If you include cities and toponyms from within California (SF, SJ, Los Angeles, SD, SV, Hollywood, Palo Alto, Menlo Park, Stanford, etc.), there are well north of 30k mentions of the state or regions within it.
10 most mentioned: ME (22,890), IA (19,353), CA (10,839), NY (9,009, including NYC matches as they're within the state), IN (6,151), TX (5,603), OH (5,085), FL (2,556), VA (2,369, excluding WV).
10 least mentioned: SD (93), RI (98), NH (136), ND (150), NE (159), WV (161), SC (178), WY (209), NM (213), VT (229).
Data (with additional cities/regions or confounding searches listed for some cases):
Alabama: 372
Alaska: 1,127
Arizona: 914
Arkansas: 264
California: 10,839 (Cities/regions: San Francisco: 7,221, San Jose: 493, Los Angeles: 1,540, San Diego: 732, Sacramento: 187, Silicon Valley: 13,782, Hollywood: 3,050, Palo Alto: 833, Menlo Park: 120, Redwood City: 33, Santa Clara: 199, Santa Cruz: 96)
Connecticut: 1,369
Colorado: 1,633
Delaware: 612
Florida: 2,556
Georgia: 1,350
Hawaii: 883
Idaho: 274
Illinois: 789 (Chicago: 3,634)
Indiana: 6,151
Iowa: 19,353
Kansas: 573
Kentucky: 301
Louisiana: 294
Maine: 22,890
Maryland: 921
Massachusetts: 620
Michigan: 1,456
Minnesota: 530
Missouri: 314
Mississippi: 248
Montana: 486
Nebraska: 159
Nevada: 580 (Las Vegas: 582, Reno: 84)
New Hampshire: 136
New Jersey: 491
New Mexico: 213 (Albequerque: 48, Los Alamos: 89)
New York: 9,009 (New York City: 1,905, Manhattan: 2,355, Wall Street: 2,934)
North Carolina: 368
North Dakota: 150
Ohio: 5,085
Oklahoma: 277
Oregon: 911
Pennsylvania: 358
Rhode Island: 98
South Carolina: 178
South Dakota: 93
Tennessee: 287
Texas: 5,603
Utah: 1,005
Vermont: 229
Virginia: 2,530 (less West Virginia: 161)
Washington: 22,246 (though many confounding matches: Washington Post: 18,634, Washington + state: 461, Washington DC: 369, Washington D.C.: 112. Seattle: 4,958, Redmond: 892.) Actual is closer to 461.
West Virginia: 161
Wisconsin: 387
Wyoming: 209
For anyone still watching, the crawl's just completed. I'm processing stats now.
Currently looking at:
- Most mentioned states (may add territories).
- Most mentioned cities (330 or so largest US for now, may add selected global cities). .
- Most mentioned countries / country identifiers, including EU.
Also looking at submitters, submitted sites, associated votes and comments, and trends over time, though those may take a bit longer to complete.
Considering companies and universities as well.
Analysis is largely awk, grep, sort, uniq --- standard shell tool stuff. Amazingly powerful though really. Crawl via wget which performed wonderfully as usual. Thanks to HN/YC for not throttling too aggressively.
And, after looking again at HN's API and determining that front pages aren't represented in it ... I'm crawling the 5,939 front pages from 20 Feb 2006 to 25 May 2023 and scraping those to find mentions of states on the actual front pages.
Aggressive crawls result in 403 after a few fetches, so my current attempt has an apparently-adequate delay parameter set. It'll take a day or two for the crawl to complete. Presently up through 22 Sept 2007.
State-name matches (with some false positives) in that set:
Delaware corporate franchise tax is due today (March 1st) (March 1, 2007)
An Early Stage Entrepreneur's Guide to New York City (March 5, 2007)
Web 2.0 Expo 2007, April 15-18, 2007, San Francisco, California (March 10, 2007)
Zillow Becomes Illegal in Arizona (April 16, 2007)
New York Times on the social news uproar (May 3, 2007)
What Silicon Valley Could Learn from Columbus, Ohio (May 22, 2007)
New York Times Will Lower Editorial Standards Online And Reduce Size Of Print Newspaper (May 26, 2007)
eHarmony sued in California for excluding gays (June 1, 2007)
New York Times versus Digg (June 6, 2007)
A Patent Lie - New York Times (June 9, 2007)
Interactive Colorado Nightlife Guide - "Social Nightworking" (June 16, 2007)
In the state of Hawaii, tax breaks make tech investments nearly risk-free? (June 27, 2007)
Top Web Apps in Arizona (June 28, 2007)
Millionaires Cash Out Of California (July 16, 2007)
C.E.O. Libraries Reveal Keys to Success - New York Times (July 21, 2007)
Pics of Web Company HQs in California - Digg, Facebook, OpenDNS.. (July 25, 2007)
Hackers find serious problems in California voting machines (July 30, 2007)
A Mystery Solved: "Fake Steve" Blogger Comes Clean - New York Times (Aug 5, 2007)
Colorado Startup Scene (Aug 5, 2007)
New York Times Sees Sense: Paywall Comes Crashing Down (Aug 7, 2007)
Parsing Miss South Carolina's Statement (Sept 1, 2007)
Texas Startup Says It Has Batteries Beat (Sept 5, 2007)
Spider-like vessel hits New York waters - can cross the Atlantic on one load of diesel fuel (Sept 8, 2007)
New York Times Launches Facebook App (Sept 12, 2007)
Theory Girl by the University of Washington CSE Band [mp3] (Sept 15, 2007)
In the event anyone wants to suggest analyses to perform, I'm game.
For now, I'm parsing out title, points, comments, submitter, and the submission site.
I'm classifying by state and city (list of 330 largest cities in the US by population), as well as a few "false positive" categories (e.g., "Washington Post", "New York Times").
One interesting angle I'm looking at is the average votes and comments by site within the data --- I'd been curious for a while as to which domains seem to be most highly considered (or at least successful in garnering votes and comments). I'm still waiting for the full archive to get pulled in, but early results are ... interesting. I'll leave it there. (Paulgraham.com rather predictably does well, though it's not the highest-rated.)
I'm also thinking of ways to do trending over time. In early data (I'm up to late 2011 / early 2012 as I'm writing this), sites including Quora and even jwz.com appear and ... do well. plus.google.com hasn't yet appeared, and a few others that come to mind don't seem represented yet. Top sites by year or over a five-year interval seems potentially illuminating.
Oh, and as far as states go, when restricted to front-page submissions only, California does far better than the Algolia search results (all non-dead/killed submissions) indicate.
Dates covered: 2007-2-20 -- 2023-5-25
Total stories: 178,072 (Should match 5,939 days * 30 stories, seems to come a bit short with 98 fewer stories than expected.)
- States mentioned: 50 (mentions: 1,344)
- Cities mentioned: 109 (checked against list of 330 largest US cities, plus a few additions of my own).
- Sites submitted: 52,687
Top 20 states by raw string-match (this is adjusted below for some overstatements):
1. new york: 316
2. california: 308
3. texas: 98
4. washington: 73
5. colorado: 38
6. florida: 38
7. arizona: 28
8. georgia: 24
9. utah: 24
10. kansas: 23
11. virginia: 23
12. oregon: 22
13. michigan: 19
14. minnesota: 16
15. north carolina: 16
16. ohio: 16
17. alaska: 15
18. indiana: 15
19. hawaii: 14
20. maine: 14
"New York" frequently appears in reference to newspapers or cities:
new york times: 98
new york city: 55
"Washington" is even harder to disambiguate, as it may refer to the state, city (DC), as a topnym for the US / US Government, etc., It's challenging to identify these textually, though the newspaper and monument mentions can be identified:
washington post: 16
washington times: 1
washington monument: 1
California is often mentioned by toponym (e.g., "silicon valley", region (bay area), or specific cities (SF, LA, etc.). I'm NOT going to adjust for these but do note the following major occurences:
silicon valley: 399
bay area: 65
california ave: 2
Adjusted top-5 state rankings:
1. california: 308
2. new york: 163
3. texas: 98
4. washington: 55
5. colorado: 38
The 10 least-represented states:
41. new mexico: 6
42. alabama: 5
43. idaho: 5
44. north dakota: 5
45. connecticut: 4
46. nebraska: 4
47. new hampshire: 4
48. rhode island: 3
49. south dakota: 3
50. south carolina: 1
1. new york: 316
2. san francisco: 192
3. boston: 117
4. seattle: 87
5. hollywood: 76
5. tempe: 69
6. berkeley: 63
7. chicago: 62
8. cambridge: 50
9. detroit: 41
10. los angeles: 30
I've also got a list of most-featured front-page sites, with the ten most frequent being: nytimes.com, techcrunch.com, arstechnica.com, bloomberg.com, wired.com, wsj.com, youtube.com, wikipedia.org, BBC (as both bbc.com and bbc.co.uk), and theguardian.com. The mean votes and comments for each are interesting, with Bloomberg topping by both average vote (255.46) and comments (178.04), from the top-10. (There are obviously sites with much higher mean scores but less frequent appearance, including singletons).
I've broken submitters by freqency, points, and comments as well, with 43,745 submitters accounting for the stories analised. The long-absent nickb tops front-page appearances at 1,322, followed by ingve (1,177) and tosh (679). I've apparently landed 60 front-page submissions averaging 329.00 points and 147.52 comments (both fairly respectable --- I'm apparently not submitting total tosh).
I'm still updating my code and checking data (and found a couple of glitches writing out the above, e.g., whatever I'd done to tabulate city mentions didn't work). Data exist presently in flatfiles, I'm thinking of dumping to sqlite for possibly easier/faster analysis.
A rough and preliminary look at countries represented in front-page titles, again with numerous issues:
1. US: 1350
2. U.S.: 1073
3. China: 634
4. Japan: 526
5. India: 477
6. UK: 288
7. EU: 225
8. Russia: 221
9. Germany: 165
10. Canada: 162
11. Australia: 157
12. Korea: 140
13. France: 116
14. Iran: 91
15. Dutch: 80
16. United States: 75
17. Brazil: 69
18. North Korea: 69
19. Sweden: 68
20. Cuba: 67
21. Egypt: 60
22. Norway: 60
23. USA: 59
24. E.U.: 54
25. Israel: 53
26. Iceland: 52
27. New Zealand: 47
28. Mexico: 45
29. Spain: 43
30. Finland: 39
31. Italy: 38
32. South Korea: 38
33. Switzerland: 38
34. Singapore: 36
35. Mali: 35
36. Chile: 33
37. Turkey: 33
38. Argentina: 27
39. Netherlands: 25
40. Syria: 25
Analysis is largely awk, grep, sort, uniq --- standard shell tool stuff. Amazingly powerful though really. Crawl via wget which performed wonderfully as usual. Thanks to HN/YC for not throttling too aggressively.
One question answered: there are in fact not 30 stories per summary, consistently.
5923 30
3 26
3 27
3 28
2 29
1 0
1 17
1 19
1 20
1 25
2007-03-10: 29
2007-03-24: 26
2007-03-25: 25
2007-05-19: 27
2007-05-26: 26
2007-05-28: 29
2007-06-02: 19
2007-06-16: 28
2007-06-23: 17
2007-06-24: 28
2007-06-30: 20
2007-07-01: 28
2007-07-07: 27
2007-07-15: 26
2007-07-28: 27
2014-01-06: 0
Spot checks suggest the others were in fact days with fewer-than-normal stories, e.g., <https://news.ycombinator.com/front?day=2007-06-30>
Comment was deleted :(
I'd long wondered about the day-of-week effect on HN. Turns out that yes, there is one.
Looking at just 2022 (most recent full-year data:
2022:
DoW Stories Points ( mean ) Comments ( mean )
Monday 1560 481368 ( 308.57 ) 288928 ( 185.21 )
Tuesday 1560 528265 ( 338.63 ) 309799 ( 198.59 )
Wednesday 1560 506977 ( 324.99 ) 310698 ( 199.17 )
Thursday 1560 514046 ( 329.52 ) 320883 ( 205.69 )
Friday 1560 465283 ( 298.26 ) 287805 ( 184.49 )
Saturday 1590 401207 ( 252.33 ) 228966 ( 144.00 )
Sunday 1560 410879 ( 263.38 ) 239186 ( 153.32 )
On the other hand, if you want a chance of your submission making the front page, weekends are probably a better bet. (My data don't show this, mind, as they don't show all submissions for a day, but this tends to support my hunch / general sense in my own submissions.)
It's interesting to note that Tuesday and Wednesday have generally captured the top spot, but there was a notable fall-off in Wednesday's standing in the Covid era. Not entirely unprecedented (Wed. lagged Tues. by 10 votes (points) in 2014, for example), but it catches my eye.
There's also the long-term trends in votes and comments over the years:
Year Stories Points ( mean ) Comments ( mean )
2007 9382 92264 ( 9.83 ) 61207 ( 6.52 )
2008 10980 294775 ( 26.85 ) 186339 ( 16.97 )
2009 10950 608603 ( 55.58 ) 303962 ( 27.76 )
2010 10950 1062763 ( 97.06 ) 491718 ( 44.91 )
2011 10949 1657004 ( 151.34 ) 632724 ( 57.79 )
2012 10980 1829402 ( 166.61 ) 778634 ( 70.91 )
2013 10950 2132819 ( 194.78 ) 998387 ( 91.18 )
2014 10905 2057628 ( 188.69 ) 916438 ( 84.04 )
2015 10950 2001269 ( 182.76 ) 845719 ( 77.23 )
2016 10977 2521394 ( 229.70 ) 1137575 ( 103.63 )
2017 10950 2776064 ( 253.52 ) 1259899 ( 115.06 )
2018 10950 2762928 ( 252.32 ) 1262654 ( 115.31 )
2019 10950 3051011 ( 278.63 ) 1447141 ( 132.16 )
2020 10980 3338150 ( 304.02 ) 1734703 ( 157.99 )
2021 10950 3376829 ( 308.39 ) 1859933 ( 169.86 )
2022 10950 3308025 ( 302.10 ) 1986265 ( 181.39 )
2023 4350 1309865 ( 301.12 ) 737425 ( 169.52 )Principality of Sealand?
1,477 results, just edging out Michigan (see my earlier comment).
New York ?
The actual difference is almost certainly moot at the end of the day since the ISPs are all in bed with the feds, but the concept of getting my internet from the same institution (read: Government) that brought us the patriot act sounds less than ideal.
You're thinking about the federal government. This is about local governments developing last mile networks for the benefit of their residents.
I'm personally skeptical of any level of government, but then I come from the place where Steven Seagal was given access to a sheriff's department tank (provided by the federal government, naturally), and drove it through a gate targeting alleged cockfighting [0]. There's plenty of horrible stuff Joe Arpaio did, but if county-level governments are receiving "tools" (including stingrays and other surveillance equipment) from the federal government, I don't see any reason they wouldn't just walk into the local government ISP whenever they wanted.
Not that I trust Cox et al much more, but I have at least passing sympathy for GP's overall point.
https://www.npr.org/sections/thetwo-way/2011/03/23/134803230...
I would agree with you except I'm still paying $90/mo for 3Mb (yes, meg) DSL from the only copper provider in the area and have been hearing from them for nearly a decade that 'fiber is coming' soon. Every time someone makes progress on providing an alternative (the local electric company has tried most recently) they have been fought HARD by the local phone company. I've even looked at renting an office in the next town over to set up my own point to point wireless solution after the local phone company quoted me $5000 per MONTH for a 10Mb service.
The competition situation is a different subject, and yes it's terrible. If municipal broadband had the effect of undoing the monopolies (as opposed to crowding out competition) it would definitely be a positive.
> (as opposed to crowding out competition)
How many places are you aware of in the US that have legitimate broadband competition? As in two or more independent providers offering gigabit or greater service with comparable real world performance? Competition in the American broadband market is largely a myth.
Back when DSL mattered there was a short period of time where it was possible to order competitive DSL, but in the end it was still using the ILEC's infrastructure so service could rarely be better or cheaper.
While it has occasionally happened it's incredibly rare for two cable companies to serve the same addresses, they definitely don't go out of their way to do it because the business model usually can't make sense of a buildout unless they expect to get most of the potential customers they pass.
Likewise for commercial fiber. AT&T, Comcast, Google, Verizon, they're not building fiber networks where anyone else already did.
Municipal fiber comes in one of two flavors. There the PON services that are basically built the same way as a commercial provider but are run as nonprofits, and there are the Amsterdam Citynet or UTOPIA style open access networks where the government organization runs the last mile but then allows providers to rent access to that last mile network to access customers.
I am a fan of the latter variety, the local government runs the local infrastructure that doesn't make sense for the free market to compete in and then service providers can focus on providing competitive service without having to worry about the last mile. Users in such areas have the most competitive internet access markets seen since the dialup era.
The former variety doesn't crowd out competition either though, it comes about because there is no real competition. Never has a local government gotten in the ISP business without the incumbent provider(s) having decided to ignore their jurisdiction's problems for a long time. PON style buildouts are faster and cheaper, so it's no surprise they get selected in a lot of areas.
> Municipal fiber comes in one of two flavors. There the PON services that are basically built the same way as a commercial provider but are run as nonprofits, and there are the Amsterdam Citynet or UTOPIA style open access networks where the government organization runs the last mile but then allows providers to rent access to that last mile network to access customers.
There is a third flavour where the municipality owns a PON service, and operates it as a for-profit service, exploiting the citizens as a monopoly at profit margins way higher than it would be feasible if run as a private company.
> There is a third flavour where the municipality owns a PON service, and operates it as a for-profit service, exploiting the citizens as a monopoly at profit margins way higher than it would be feasible if run as a private company.
Would you like to provide some examples? Not saying it can't happen, it's certainly possible if maybe a little bit implausible, but I haven't heard of such situations and I would assume that if it was even remotely common that the Comcasts and AT&Ts of the world would be shouting from the rooftops about it.
Better to give access to people who don't need the Patriot Act to go through your stuff at will.
I don't think your city/county/state is administered by the feds.
Crafted by Rajat
Source Code