2021.10.24 21:52 Isvesgarad How do you test your pipelines?
Junior engineer here trying to learn how to properly test data pipelines. Currently my team just has unit tests for our Scala code, but nothing to test the full pipeline run (SNS trigger + EMR + S3 write).
What is the best way to end to end test full pipelines?
My team has around 70 different transformations with 400+ source files, so it doesn’t seem realistic to create mock data and test every transformation.
Should we just have 1 test with mock data that flows through the entire pipeline? Test every transformation that was changed in git commits? What patterns does your team follow?
submitted by Isvesgarad to dataengineering [link] [comments]
2021.10.24 21:52 mwilliams705 Who needs a license for YouTrack?
I am considering switching to YouTrack for my team but I am not sure who will need a licesnse to use the service. Our IT department is staffed by 7 employees who currently use Jira for our ticketing system. We have ~90 employees that send tickets in through emails to Jira. With YouTrack, will our employees need a license to YouTrack to put in tickets or can we stick to licensing our IT department alone and still have our users put in tickets?
submitted by mwilliams705 to Jetbrains [link] [comments]
2021.10.24 21:52 ReactQ UFC Women's Rankings Update: Joanna Jędrzejczyk has been removed from the rankings // Jessica Penne enters the Strawweight Rankings at #15 // Mackenzie Dern enters the Women's P4P list at #15
|submitted by ReactQ to WMMA [link] [comments]|
2021.10.24 21:52 That_One_Nobody_ That moment when.. 😭
|submitted by That_One_Nobody_ to SkyChildrenOfLight [link] [comments]|
2021.10.24 21:52 mobettachedda Wife got ahold of the camera today. Not bad imo. Maybe I'll get her one for Christmas lol.
|submitted by mobettachedda to SonyA6400 [link] [comments]|
2021.10.24 21:52 GameCollector98 [USA] [H] PayPal [W] Atari Jaguar Console
2021.10.24 21:52 SnooTigers5587 Caviar + Flower Recommendations?
Looking to pick up some caviar for the first time from AYR Gibsonia this week during their 30% off sale. Any recommendations on caviar? As well as some flower to pair it with? I’ve been pretty happy with rhythms flower but always open to switch it up!
submitted by SnooTigers5587 to PaMedicalMarijuana [link] [comments]
2021.10.24 21:52 VanderBacon Alpha Legion decal
Some time ago I asked for help with Alpha Legion decals. With that and the help of some friends and Google I came with this LINK. Love to hear what you think! Am I missing something or do some icons just not belong on this sheet? I'm not that into the legion's lore..
submitted by VanderBacon to alphalegion [link] [comments]
2021.10.24 21:52 theOnetheOnly_mrCool End of series
2021.10.24 21:52 yeah_deal_with_it Can I get a sneaky profile review please?
|submitted by yeah_deal_with_it to hingeapp [link] [comments]|
2021.10.24 21:52 PyoobyCare THEY CAN'T DO ANYTHING... THEY HAVE NO RIGHTS!!!!!
2021.10.24 21:52 IceDogBL Gluttony Question
If gluttony (in regards to food and drink) is the partaking in consumption purely for pleasure and not out of necessity, wouldn’t eating candy be gluttony in most cases, even if you don’t eat much at all?
Maybe my understanding of gluttony is flawed.
What constitutes the venial sin of gluttony?
Thanks a ton!
submitted by IceDogBL to Catholicism [link] [comments]
2021.10.24 21:52 wendybird242 I guess this is for adults?
2021.10.24 21:52 YL0000 Sourdough troubleshooting
I made a wholewheat sourdough for the first time. The gas bubbles are small and the bread is heavy and dense. The dough did rise a lot during the bulk fermentation stage but it began to have a soya sauce smell at the end of it. Does this suggest it was over-fermented? The dough didn't grow much in the second proof. The dough was then baked in a le Creuset pot in a small 33L oven.
The recipe I followed is from Peter Reinhart's The Bread Baker's Apprentice. I adapted his recipe of Poilane-Style Miche to a small size for my small oven. More specifically, I used
43.7g barm, 56.2 g whole wheat flour and 24.9 water for the firm starter
200g whole wheat flour, 5g salt, 124.8g firm starter as above and 112.5g water.
Does anyone see quickly what is possibly wrong here? I suspect that there was not enough hydration. Are there different types of whole wheat flour? Are some more difficult to operate than others? The flour I used was Bob's Red Mill organic whole wheat flour. I kneaded it by hand for 20ish minutes but wasn't able to make a windowpane. Is it because that I did not knead it sufficiently? (My experience was that 10 min of kneading was enough for the dough of a similar size from the regular bread flour to pass the windowpane test.)
Processing img fir4pwizshv71...
submitted by YL0000 to Sourdough [link] [comments]
2021.10.24 21:52 kittieslovelettuce Hikes for a solo woman?
New to the area, don’t have any non-work friends. Looking for hikes that are within an hour drive of Gwinnett that you feel would be safe for a solo woman. Preferably less than a 3 hour hike with good trail markings (it’s amateur hour over here).
submitted by kittieslovelettuce to Gwinnett [link] [comments]
2021.10.24 21:52 bot_neen La Línea Dorada De Ebrard De Mancera De Sheinbaum O De Slim
|submitted by bot_neen to Mexico_Videos [link] [comments]|
2021.10.24 21:52 wisemanbob1274 [WP] Suicidal person who has lost everything decides, because they have nothing to lose, become a hero to give point to a pointless life, and a person that is not afraid to die goes far.
Possible ending, The person does die, as they knew they would, this being a suicide run for a pointless life. But they die happily, having saved many and inspired more, and having a point to their life.
submitted by wisemanbob1274 to WritingPrompts [link] [comments]
2021.10.24 21:52 Mini_Nugget69 yummy
|submitted by Mini_Nugget69 to brookemonknudephotos [link] [comments]|
2021.10.24 21:52 Arun_Samra1 Teen heaven 💋😈
|submitted by Arun_Samra1 to noelle_leyva [link] [comments]|
2021.10.24 21:52 IndigoIshtar MRU is accepting religious and medical exemptions until October 31, 2021.
|submitted by IndigoIshtar to MRU [link] [comments]|
2021.10.24 21:52 bah_bouh A Complete Masterclass of Virtualization from scratch - CURSUSA
|submitted by bah_bouh to udemyfreebies [link] [comments]|
2021.10.24 21:52 bhonduno1 Lavish wedding proposals are cringey
Planning to spend life together with someone shouldn't be a surprise. It's a mutual decision taken together over a period of time. Lavish wedding proposals come off as ostentatious.
submitted by bhonduno1 to unpopularopinion [link] [comments]
2021.10.24 21:52 DthDrp4Dyz Offering: English (Native) Seeking: Russian.
I know the Russian Alphabet and can read some words, I need to build my vocabulary and learn the cases. Would also like to gain some Russian friends.
submitted by DthDrp4Dyz to language_exchange [link] [comments]
2021.10.24 21:52 MNIMDMCOTAOTNGOTFL I just want to be left alone
Seriously, leave me the fuck alone. I don't give a fuck about politics or communist LARPING or whatever. I am just a normal person. I did everything I was supposed to do. Wasted almost two decades of my life in 'education', got a masters degree in STEM and still I can't even get a minimum wage job. The best I can seem to do is a part-time exploitative retail job. It is so fucking degrading to have to stand around all day and labour at the click of someone's fingers. I don't think I am better than anyone else for having went to university, I am just mad that I was lied to from the very start.
All I want to do is build a fucking log cabin on a bit of land. That's not at all possible in my country. Bullshit planning permission laws and extortionate land prices mean I would have to slave away for decades to be able to afford to do such a thing. The only purpose of the 'law' is to trap you into this bullshit system.
Who the fuck has the right to tell me I can't build a log cabin on a bit of land in my own country? There are people who own thousands of acres of land in my country and they do fuck all with it. Nobody can use it. Just these greedy fucks have their names on a bit of paper so apparently now nobody can use that land or even go on it. I think we are only allowed on something like 10% of land in this entire country.
Now my bullshit government is trying to outlaw living in vans/ caravans. They are tightening the noose. It is even illegal to camp out in a fucking tent, or light a god damn fire to stay warm. Who the fuck do these paedophilic bastards think they are?
I am 24 years old and forced to live in my parents' tiny home. There are two rooms downstairs and three rooms upstairs. Five people live in the house and so I basically have to spend all of my time in my tiny box of a bedroom. This is no way to fucking live. I dread another winter of this. The government is going to 'lock us down' this winter again. Again, who the fuck do they think they are? Might be okay being locked down in your mansion with swimming pools, gardens, lavish rooms etc. But you try spending months on end in a fucking box. I hate these cunts. I hate my life.
submitted by MNIMDMCOTAOTNGOTFL to antiwork [link] [comments]
2021.10.24 21:52 Detrimentos_ [Redacted]