Free Range Internet outage: Jan 9 - Jan 10, 2025
Jan 9 12:59pm
A long-haul fiber cable owned by Lumen (CenturyLink) is cut during grading activities at a construction site in Goodyear, AZ.
This cable connects our Gila Bend microwave relay to a datacenter in Phoenix, where we access wholesale internet.
Free Range Internet and other providers go offline.
Cellular services from AT&T are also affected.
Jan 9 1:02pm
Free Range Internet identifies the issue as being upstream and opens a ticket with Lumen.
At the time of the outage, the on-call technician is replacing KZAO's FM transmitter on A mountain. He returns to the office as soon as possible.
Jan 9 1:19pm
Posted on Facebook (AJOmundo, Ajo Classifieds, Free Range Internet's page)
Free Range Internet is currently experiencing an outage due to a problem with our upstream provider.
We do not have an ETA at this time.
Jan 9 1:45pm
All Free Range Internet staff is present at the office.
A plan to connect local businesses to a limited-capacity backup satellite link is discussed.
We are reluctant to enact the plan because it could trigger a different outage, effectively making the current outage unnecessarily longer.
No other information about the outage is known at that time.
Jan 9 2:12pm
From Lumen
On January 09, 2025 at 19:59 GMT, Lumen discovered a service impact in Goodyear, AZ.
Your customer trouble ticket has been related to a network event; all network events are automatically escalated to Lumen leadership.
Jan 9 2:19pm
Observation
Due to the location (Goodyear), we understand that the outage is very likely a fiber cut, which would take several hours to fix or more.
Free Range Internet decides to enact the satellite back-up plan for businesses.
Jan 9 2:44pm
Posted on Facebook
Free Range Internet is currently experiencing an outage due to a fiber issue with our upstream provider in Goodyear.
We do not have an ETA at this time.
Business and commercial accounts in Ajo are now connected to a backup system.
Jan 9 3:15pm
From Lumen
The Lumen Network Operations Center (NOC) was engaged to investigate the failure. Initial investigations suggested a fiber fault to be the source of the service interruption. As such, the Lumen NOC engaged Field Operations for dispatch to a facility to assist with isolation efforts. After arriving onsite, Field Operations measured light levels and observed a fault approximately 19 miles from the facility. A splice team has been engaged for dispatch to the suspected damage location to assist with restoration efforts.
Jan 9 3:55pm
Free Range Internet starts upgrading software on core routing equipment, to avoid having to create an outage to conduct the upgrades another day.
Jan 9 4:12pm
From Lumen
Field Operations is onsite and has located the fiber cut, caused by grading at a construction site, and are currently assessing the damage to determine the repair plan. The damage has been found to be impacting 60, 96, 216, and 288-count fiber cables.
Jan 9 4:12pm
Lumen sends Free Range Internet a picture of the damage to the fiber optic cables.
Free Range Internet determines the damage is severe.
We start looking at higher-capacity backup options.
Jan 9 4:36pm
Text Message
FREE RANGE INTERNET
We are experiencing an outage due to a fiber cut in Goodyear, caused by grading at a construction site.
The damage is severe and impacts several interstate fiber cables.
We do not have an ETA at this time, but we are confident this will be resolved before tomorrow morning.
Thank you
Alex at Free Range Internet
Jan 9 4:39pm
Posted on Facebook
Free Range Internet is currently experiencing an outage since 12:59:24 PM due to a fiber issue with our upstream provider in Goodyear, caused by grading at a construction site.
The damage is severe and impacts several interstate fiber cables.
Jan 9 4:46pm
From Lumen
A construction team has been engaged for dispatch to the damage location and is expected to arrive by 01:00 GMT on January 10, 2025. The next update will be provided by 02:00 GMT, or as new information becomes available.
Jan 9 6:57pm
From Lumen
The construction team is about 15mins away from the location now. Once onsite they will begin efforts to expose the damaged fiber and duct. It is expected that approximately 2000 feet of fiber will need to be replaced for each of the four cables, and approximately 200 feet of trenching as the damage pulled the fiber cables from the manhole.
Jan 9 7:26pm
From Lumen
The construction team and the splice team have both arrived onsite now, and the construction team is preparing to begin excavation efforts.
Jan 9 7:31pm
Text Message
FREE RANGE INTERNET
Our upstream provider, CenturyLink, has informed us that they need to replace 2000ft of fiber cable in Goodyear.
We expect this outage to extend well into the night.
Jan 9 7:43pm
Triad Wireless, who is a wholesale Free Range Internet customer in Gila Bend, reaches out about enabling a high-capacity backup using an alternate fiber route they have access to.
A called is scheduled ASAP.
Jan 9 8:00pm
Free Range Internet investigates the possibility of restoring Wi-Fi calling capabilities to all customers using the low-capacity backup option.
Jan 9 8:42pm
Free Range Internet enables Wi-Fi calling capabilities to all customers using the low-capacity backup option.
However, we are unable to assess how effective this measure is.
Jan 9 8:50pm
Posted on Facebook
We have enabled Wi-Fi calling over a backup link.
Customers using AT&T or T-Mobile should now be able to place calls when they are connected to Wi-Fi.
Jan 9 8:59pm
From Lumen
The construction team continues efforts to expose the damaged fiber and has found that all four ducts have been damaged as well as all the spare ducts.
Jan 9 9:41pm
Staff at Free Range Internet and Triad Wireless get on a call and refine their high-capacity backup plan, and will meet up in Gila Bend to implement it.
Jan 9 9:57pm
From Lumen
Efforts to excavate the damaged conduit and ducts remains ongoing.
Jan 9 9:59pm
Text Message
FRI Update
The damage is greater than expected, and we do not know when service will be restored. We are setting up a temporary wireless backup between Gila Bend and Phoenix.
If you work from home or depend on internet access, please provide some form of documentation to this number.
We will make every effort to include you in the backup network as soon as it becomes available.
Jan 9 10:25pm
From Lumen
The ducts have been repaired and the Splicing Team have commenced pulling the 72 and 96-count fiber cables through the ducts. Once the 2 cables are pulled in the technicians will begin pulling the 216 and the 288 count cables.
Jan 9 10:45pm
Free Range Internet and Triad Wireless staff meet in Gila Bend and begin executing the high-capacity backup plan.
Jan 10 12:14am
From Lumen
The Splicing Team continues to pull the replacement cable through the ducts.
Jan 10 12:51am
The high-capacity backup link is now online.
Free Range Internet staff head back to Ajo.
Jan 10 1:07am
From Lumen
The 72 and 96-count cables have been pulled through the ducts and splicing has commenced on the 72-count cable. While Splicing is ongoing, technicians are working to pull the 216 and 288 count cables through the ducts.
Jan 10 1:41am
Free Range Staff is back in Ajo, transfers businesses to the new high-capacity backup and begins processing backup connection requests from residential customers.
We also make configuration changes to our routers to ensure service will come back on automatically for everyone when the upstream fiber is fixed.
Jan 10 1:42am
From Lumen
Splicing efforts are ongoing for the 72-count and technicians are preparing the 216-count fiber for fusion. Additional field personnel are working to pull in 4000 feet of replacement fiber through multiple hand holes to replace the 288-count fiber cable.
Jan 10 3:02am
From Lumen
Splicing has been completed on the 72-count cable and efforts continue to pull the 288-count fiber through the ducts.
Jan 10 3:34am
Free Range Internet Staff is done processing backup connection requests from residential customers.
All customers who need internet for work or health reasons now have internet access.
Jan 10 3:38am
Posted on Facebook
IF YOU NEED INTERNET, REACH OUT
520 333 7507
hello@ajofreerange.com
1440 N 2nd Ave
ETA for repair is 6am-3pm Friday Jan 10
Jan 10 4:23am
The fiber cut is fixed by Lumen technicians.
Service is restored to 98% of Free Range Internet customers within 1 minute.
Jan 10 4:26am
From Lumen
Splicing efforts remain ongoing and the NOC is beginning to see some alarms clearing.
Jan 10 5:28am
From Lumen
Splicing efforts on the 72 and the 92-count fibers continue and cable placement remains on the 288-count cable.
Jan 10 6:57am
From Lumen
Repair crews have completed splicing the 72-count cable. Teams continue splicing the 216-count cable. As teams have exceeded the mandatory safe working hours, additional teams have been engaged to dispatch to the fault site to complete placing the 96 and 288-count cable. The ETA is 2 hours.
Jan 10 7:58am
From Lumen
Splice crews continue splicing the 216-count cable.
Jan 10 9:00am
Over 10 customers are still experiencing issues adjacent to the outage.
Reset routers, TVs won't connect, etc.
Free Range Internet staff is dispatched to help on a case-by-case basis.
Jan 10 9:15am
From Lumen
The NOC has engaged repair crews for an update on repair progress.
Jan 10 10:12am
From Lumen
Splicing is ongoing currently and the Lumen NOC has observed service affecting alarms continuing to clear.
Jan 10 10:56am
From Lumen
Crews found additional damage on the 288-count cable. Crews continue pulling the replacement cable to the last manhole. Splicers continue splicing the 216- and 96-count cable.
Jan 10 10:57am
Text Message
Free Range Internet:
Service was restored at 4:23AM.
Thank you for your patience.
We will publish a detailed report at 5pm at https://ajofreerange.com
Jan 10 12:43pm
From Lumen
Splicing has been completed on the 72 and 216-count cable. Splicers continue to splice the 96-count cable and crews have placed the 288-count and prepping both ends for splicing.
Jan 10 1:42pm
From Lumen
288-count cable has been prepped and splicing has commenced.
Jan 10 2:57pm
From Lumen
Splicing is ongoing and the NOC has observed additional alarms beginning to clear.
Jan 10 4:30pm
The ajofreerange.com website is updated to reflect that the detailed report time is pushed to 8pm.
Jan 10 4:39pm
From Lumen
Splicing operations are approximately halfway completed on both the 288-count and 96-count fibers. Crews are awaiting relief at the west hole to finish the 96-count fiber. Splicing continues on the 288-count fiber and the east side of the 96-count fiber.
Jan 10 5:30pm
From Lumen
Repair crews are continuing to make progress with splicing operations at both ends of the fiber.
Jan 10 6:39pm
From Lumen
Splicing is still in progress, and the Lumen NOC is monitoring stability and have determined that some of the service-affecting alarms have cleared.
Jan 10 7:30pm
As of 7:30pm, Lumen is still replacing the damaged cables.