October 18, 2018, 07:42:53 pm
News: If someone gives you assistance give them a Good Star
Pages: 1 ... 6 7 [8] 9 10 ... 14   Go Down
Author Topic: Post race ... Continued ... Again  (Read 17643 times)
Jason Reed
Administrator
*

Stars: +4/-0
Offline Offline

Gender: Male
Posts: 4487


Pure Evil Administrator


WWW
« Reply #105 on: December 07, 2017, 03:12:12 pm »


I really hate this time of year due to gift giving (and feeling that there is an imbalance between my siblings and I)... We can do the whole "thank you for the gift card, here is a gift card" which just feels silly (although when I receive a gift card I do make sure to 1. spend it on a something I might not otherwise buy for myself and 2. let the gifter know what they got me. Being that my siblings don't do the same, I don't have any feed back that they actually liked the gift. I'd like to get them actual gifts, but we all buy the things we want, for ourselves, when we want them Another Tounge (and I don't really know them well enough to know what to get or what they have). I'd like to say "can we just not do gifts this year?" but don't want to sound like a cheapskate or jerk sofa
Yeah, gift giving doesn't come to me naturally. I mean one of our first Christmases together I bought her pots and pans doh. Funny enough she stayed with me Wink I've gotten better though, but it's still not easy to get everything together, and hope that I'm giving her as much as she gives me. Thankfully I don't have siblings and my mother doesn't celebrate so I don't have to worry about it, just the wife, and three boys Grin

doing anything or just relaxing?

Sadly I'll probably be doing everything not to get sick. It's been a while since I've been really sick, and the last couple of weeks have been stressful, and the next two aren't going to be much better, so I'm prime for a nasty one. So I'll be fighting it every step of the way.

Sounds annoying. Looks like my Sanwa does 6 lines, I've never tried to modify it though.

I have to look maybe I can make it work better, I just have to remember that when I'm viewing a map I shouldn't use the scroll wheel.

Reading that makes me feel pretty fortunate that I don't have to worry about bandwidth.
The problem is that we don't have the fastest dsl available, usually because we wouldn't use it. We don't stream a lot of videos or tv. The thing is that technology eventually assumes that you'll have the bandwidth and just takes what it wants. It's kind of like how software became very bloated and bad at memory management once it was normal for more people to have more memory.

Hopefully it will come naturally in time, with practice yes
Yeah, I just need to get good enough where the project manager actually uses me. Working in SAP when it is our development environment is nothing like working in a live system. That's where you get your real experience.

What about when you first came to Germany?
Exactly, I've done it many times where I visited places where I didn't speak the language. The difference is that most of the time Germans will actually try to speak English. The French are not known to be like that. Of course that's why there's Google Translate Wink

And a rough two weeks it has been sweat I have gotten more used to my phone at least Another Tounge

So... I have a script that deletes a certain chunk of client's data that I use on the rare occasion it gets corrupted (typically when our system is fed via an api). Well, this new client was hammering our system and the sql locks were resulting in corrupt data. I didn't want to wipe the entire profile, only about 10% though, so I looked up the id I needed then modified my script... and wiped the whole.ducking.table...

1pm: I didn't realize my mistake till another client called in... at which point I scrambled to set up a second database to restore a backup. I didn't want to do a full restore over top of our existing data as we would lose teh data that came into other tables. The next issue was that I simply didn't have room to create a second database... So I contacted our host (my hope was they could "loan" us a server) and they suggested we pop on a 2tb external drive (I didn't know they made them that large).
5pm: The restore finally got started, I stayed at the office, getting 30-60 minute updates "5%" "10%" till I decided there really wasn't anything I could do, so I went home and ate around 9pm.
10pm: I calculated that it should be done at roughly 2-5am, and that I should try to sleep... but that wasn't easy to do. I was receiving half hourly updates
2am: (as expected) I received a "we're at 92%, I'll update you shortly" but then I didn't hear back for over an hour... so my mind started telling me all kinds of things went wrong... so I called them at about 3-4am. It turns out that when a restore hits 100%, it does some stuff, and in the case of my database, this step can take an hour...
4am: don't try to open the error log file, on the server, if it's huge... the tech crashed sql server and when it came up it was no longer in "restore" mode, but when we tried to bring it online... it said that the restore was interrupted... I have to assume due to that crash... so let's start the process over again...
7-8am: I went to teh office as usual, not sure if I would even have a job (of course they aren't going to fire the one guy that can restore the data before he restores it) after I get all the data back... I knew it would be at least noon before the current restore would be ready, so I tried to rebuild some of our more important (read: complainers) clients data from various log tables... unfortunately that wasn't possible because i needed some database id's that were only logged if the client did an update.
1pm: restore is at 100%, now for the unknown amount of time till it goes from 100% to complete (turns out there is a flag that could have been set that would give a % on this step, but the tech didn't set it and/or was unaware of it)
2pm: restore is back up and 10 minutes later I have imported the table I wiped. I checked the logs and there was no new data inserted, so we were totally back online and all good!

(pretty sure my job is safe, later that day, or the next, we had a meeting regarding future projects sofa )

Good to hear that you got it fixed. Large database systems always scare me :couch:

Sounds to me that the APIs and/or the database handling needs some optimization if a customer slams it and it has problems, is it some sort of threading problem?
Logged

-- signature --

msimmons
Administrator
*

Stars: +3/-0
Offline Offline

Gender: Male
Posts: 8501



WWW
« Reply #106 on: December 07, 2017, 07:03:54 pm »

Yeah, gift giving doesn't come to me naturally. I mean one of our first Christmases together I bought her pots and pans doh. Funny enough she stayed with me Wink
heh, depending on the situation I can see pots & pans being a good gift laugh1


I've gotten better though, but it's still not easy to get everything together, and hope that I'm giving her as much as she gives me. Thankfully I don't have siblings and my mother doesn't celebrate so I don't have to worry about it, just the wife, and three boys Grin
And you probably have a good idea of what they would like, I can't think of anything one of my sisters would be interested in, and the other sister most likely has everything she would want in relation to the interests I know of...


The problem is that we don't have the fastest dsl available, usually because we wouldn't use it. We don't stream a lot of videos or tv. The thing is that technology eventually assumes that you'll have the bandwidth and just takes what it wants. It's kind of like how software became very bloated and bad at memory management once it was normal for more people to have more memory.
Yeah, I worry about my phone data on occasion. When I'm on wifi I don't care what it's doing, but I don't know if it's smart enough to not eat up my "minutes" doing things I don't really need.


Working in SAP when it is our development environment is nothing like working in a live system. That's where you get your real experience.
heh I read this a few times and am still confused. (If I understand correctly, your dev & prod are very different, why is that?)


Exactly, I've done it many times where I visited places where I didn't speak the language.
I totally missed the "not" in your "not the first time" doh


The difference is that most of the time Germans will actually try to speak English. The French are not known to be like that. Of course that's why there's Google Translate Wink
I've actually heard they don't want foreigners slaughtering their language so will speak other's tongues to prevent it.


Good to hear that you got it fixed. Large database systems always scare me :couch:
Damn thing is over 1tb... I'd really like the time to get rid of duplication and unused things, but that's not likely to happen. I get buried in constant requests for new features while having a backlog of big fixes (most likely due to everything being rushed).


Sounds to me that the APIs and/or the database handling needs some optimization if a customer slams it and it has problems, is it some sort of threading problem?
exactly! This database has been around for over 17 years, migrated & upgraded from server to server, maintained & modified by different people depending on the "era." I find dumb crap like tables that don't have indexes, or tables that have too many indexes (indexes are good for seeking/reading, bad for inserting), and honestly, while I do want to learn... I don't want to learn... I just want to be a programmer Another Tounge When I say "we should have a DBA give the database a once over, to correct some of these glaring issues" I receive "well, we could hire someone, or y'all can have an xmas bonus." So duck it.

I guess if I ever get a second to breath, my first step is to diagram what the actual data flows even are... What seems to be the straw breaking the camel's back is that we have a client that uses our incoming api, which makes some updates, then pushes that data out to various sources. It's actually pushing WAY more out than what is coming in since there are not only more locations to go to, but we have "derived" values that are based on the values they update... So I may need to figure out how to throttle or adjust timings...
Logged

                 ..... .....

Jason Reed
Administrator
*

Stars: +4/-0
Offline Offline

Gender: Male
Posts: 4487


Pure Evil Administrator


WWW
« Reply #107 on: December 08, 2017, 08:56:16 pm »

heh I read this a few times and am still confused. (If I understand correctly, your dev & prod are very different, why is that?)

A dev system is pretty much static, there's data to play with and such but nothing really changes. A Prod system is alive, and dynamic. The data is constantly changing, as there could be 100's or 1000's of people working with it at the same time. You screw up something there it can affect a lot of people really fast. Oh and there's no SVN. There's no versioning in the code. You activate the code and all the old stuff is gone, and never coming back. In a dev system you screw up the programming, you have time to figure out what you did wrong, and correct it and get it all back working again. Do that on a live system and it could make your 48 hour hell this week seem like a walk in the park.

I've actually heard they don't want foreigners slaughtering their language so will speak other's tongues to prevent it.

My wife's suggestion is that it is more like that the German realizes that they probably speak better English than the English speaker speaks German and so to facilitate an easier communication the German will speak English.

Damn thing is over 1tb... I'd really like the time to get rid of duplication and unused things, but that's not likely to happen. I get buried in constant requests for new features while having a backlog of big fixes (most likely due to everything being rushed).

Yep that's a lot of data. What the heck do you do with all that data?

exactly! This database has been around for over 17 years, migrated & upgraded from server to server, maintained & modified by different people depending on the "era." I find dumb crap like tables that don't have indexes, or tables that have too many indexes (indexes are good for seeking/reading, bad for inserting), and honestly, while I do want to learn... I don't want to learn... I just want to be a programmer Another Tounge When I say "we should have a DBA give the database a once over, to correct some of these glaring issues" I receive "well, we could hire someone, or y'all can have an xmas bonus." So duck it.

How many tables are in this database? Are they all relatable? Are they all used? That would be my first check, are there any tables that are just no longer used. If they are no longer used how old is the data inside them.

I guess if I ever get a second to breath, my first step is to diagram what the actual data flows even are... What seems to be the straw breaking the camel's back is that we have a client that uses our incoming api, which makes some updates, then pushes that data out to various sources. It's actually pushing WAY more out than what is coming in since there are not only more locations to go to, but we have "derived" values that are based on the values they update... So I may need to figure out how to throttle or adjust timings...

You know there's plenty of software that can generate a diagram for you so you can see how the tables are related. That I think would be a good start, and then see how the code relates to the database tables.
Logged

-- signature --

msimmons
Administrator
*

Stars: +3/-0
Offline Offline

Gender: Male
Posts: 8501



WWW
« Reply #108 on: December 08, 2017, 09:26:37 pm »

A dev system is pretty much static, there's data to play with and such but nothing really changes. A Prod system is alive, and dynamic. The data is constantly changing, as there could be 100's or 1000's of people working with it at the same time.
Ah so only your data is different, the code is the same. I was thinking you were saying the systems didn't match. (I have a difference in mine because you can't use a credit card "swiper"" or "pin pad" in a dev environment, yet I have to develop it somehow... so doing it live...


Yep that's a lot of data. What the heck do you do with all that data?
very good question... My largest tables are full of the xml transmissions that go back & forth between apis so I can prove when things are not our fault. One of my proposals is that I move these to a separate database (on the same server) to make managing backups (and recoveries) easier.


How many tables are in this database? Are they all relatable? Are they all used? That would be my first check, are there any tables that are just no longer used. If they are no longer used how old is the data inside them.
There are a lot (50-100) tables and I would say at least 95% are in use. Think about when you are making a hotel reservation and you want to add a "package" (like breakfast), I have 12 tables for that alone.


That would be my first check, are there any tables that are just no longer used. If they are no longer used how old is the data inside them.
I think I made the is it used and can i delete this old data check a few years ago, and for a time (before we added more drives to the server) I was running a script monthly that was deleting records older than a decided date from certain tables.


You know there's plenty of software that can generate a diagram for you so you can see how the tables are related. That I think would be a good start, and then see how the code relates to the database tables.
That would be a good start! Have any suggestions (or keywords to Bing)? I feel like I tried to find something before but could not. I think one of the biggest issues is that the software has been around, and evolving, for about 20 years so there may be some underlying items that are running unnoticed, but correctly, and I don't want to disturb them... We do have a mix of languages, although I'm phasing out what I can when I can and replacing with .net.

 
Logged

                 ..... .....

Bonker Gudd
Monster Poster
*

Stars: +0/-0
Offline Offline

Gender: Male
Posts: 1285



WWW
« Reply #109 on: December 17, 2017, 01:13:21 am »

I'm preparing my autopsy.
Logged


Jason Reed
Administrator
*

Stars: +4/-0
Offline Offline

Gender: Male
Posts: 4487


Pure Evil Administrator


WWW
« Reply #110 on: December 25, 2017, 01:06:51 pm »

Merry Christmas guys.

Hope everyone is having a nice Christmas period. We're being really lazy smile
Logged

-- signature --

msimmons
Administrator
*

Stars: +3/-0
Offline Offline

Gender: Male
Posts: 8501



WWW
« Reply #111 on: December 27, 2017, 09:33:30 pm »

Hope everyone is having a nice Christmas period. We're being really lazy smile
Wasn't bad Another Tounge back at work today, I feel like I'm chasing my tail with the servers but at least they are up and running... getting new hardware next month, but I still think some software changes are in order...

trying not to get sick, it seems everyone I know has or had this horrible sickness, I keep fluctuating between feeling fine and feeling like I might be getting it sofa
Logged

                 ..... .....

Jason Reed
Administrator
*

Stars: +4/-0
Offline Offline

Gender: Male
Posts: 4487


Pure Evil Administrator


WWW
« Reply #112 on: December 28, 2017, 03:57:25 pm »

Wasn't bad Another Tounge back at work today, I feel like I'm chasing my tail with the servers but at least they are up and running... getting new hardware next month, but I still think some software changes are in order...

trying not to get sick, it seems everyone I know has or had this horrible sickness, I keep fluctuating between feeling fine and feeling like I might be getting it sofa

Gah! Hope you don't get sick then! I'm still off work until next Tuesday. Been working on some personal projects, and helping my son with his Christmas gift. We got him a project kit for the Raspberry Pi where he can build a RFID reader/writer. It even comes with a couple of RFID tags one in the form of a keychain fob, and the other in a card. After some start and stops with configuration today he's been able to read from the keyfob RFID.
Logged

-- signature --

Bonker Gudd
Monster Poster
*

Stars: +0/-0
Offline Offline

Gender: Male
Posts: 1285



WWW
« Reply #113 on: December 29, 2017, 01:10:20 am »

Obey me mf_popeanim
Logged


Jason Reed
Administrator
*

Stars: +4/-0
Offline Offline

Gender: Male
Posts: 4487


Pure Evil Administrator


WWW
« Reply #114 on: December 29, 2017, 12:14:06 pm »

Obey me mf_popeanim

No!
Logged

-- signature --

Jason Reed
Administrator
*

Stars: +4/-0
Offline Offline

Gender: Male
Posts: 4487


Pure Evil Administrator


WWW
« Reply #115 on: December 31, 2017, 11:08:58 pm »

Happy New Year!! yay

Hopefully this will turn out better. (Hope always dies last).

This weekend I took the boys to see the new Star Wars film. I enjoyed it, and so did they.

Stay safe and hope you have a great new year celebration.
Logged

-- signature --

msimmons
Administrator
*

Stars: +3/-0
Offline Offline

Gender: Male
Posts: 8501



WWW
« Reply #116 on: January 01, 2018, 05:29:51 pm »

Happy new year yay

Last year I woke up sick as duck on the 1st... this time I started early and got sick on the 28th lol... I saw star wars on the 28th, went home and took my temp, was 99F, and 100 by morning. I think I'm just about better now though.

SW TLJ was ok, not in my top 6 or anything...
Logged

                 ..... .....

msimmons
Administrator
*

Stars: +3/-0
Offline Offline

Gender: Male
Posts: 8501



WWW
« Reply #117 on: January 01, 2018, 05:33:05 pm »

Been working on some personal projects, and helping my son with his Christmas gift. We got him a project kit for the Raspberry Pi where he can build a RFID reader/writer. It even comes with a couple of RFID tags one in the form of a keychain fob, and the other in a card. After some start and stops with configuration today he's been able to read from the keyfob RFID.
dbl post because I'm too snotty/lazy to hit edit Another Tounge This sounds awesome! What are the possible projects to use this? Plans to chip the family in the future?
Logged

                 ..... .....

Jason Reed
Administrator
*

Stars: +4/-0
Offline Offline

Gender: Male
Posts: 4487


Pure Evil Administrator


WWW
« Reply #118 on: January 01, 2018, 06:06:08 pm »

dbl post because I'm too snotty/lazy to hit edit Another Tounge This sounds awesome! What are the possible projects to use this? Plans to chip the family in the future?

Actually, he got an idea for a project from watching a kid's science show. Back some years ago they produced a show describing a connected house. There was one particular bit where the son comes to his desk lays his school bag in a specific section of his desk, and starts a program on his computer. The computer tells him what books are missing from his school bag for the next day. William would like to try to build something like this for himself.

Here in Germany once the children reach 5th grade they enter basically high school. This means a lot of different classes on different days. They won't always have the same class on the same day. For example one day they may have English, Math, Physics, and Geography; then on the next day they may have English, German, Math, Informatic (Computer Sciences), and History. So they are constantly needing to repack their school bag. William has the problem that he really doesn't like packing his school bag, or he forgets to put something he needs in the bag. So he thought it might be an idea to try to build something like this, using what he's learning with the Raspberry Pi. I'm helping him out as he gets stuck, and if there's any programming things he doesn't understand.
Logged

-- signature --

msimmons
Administrator
*

Stars: +3/-0
Offline Offline

Gender: Male
Posts: 8501



WWW
« Reply #119 on: January 11, 2018, 09:47:32 pm »

Actually, he got an idea for a project from watching a kid's science show. Back some years ago they produced a show describing a connected house. There was one particular bit where the son comes to his desk lays his school bag in a specific section of his desk, and starts a program on his computer. The computer tells him what books are missing from his school bag for the next day. William would like to try to build something like this for himself.

Here in Germany once the children reach 5th grade they enter basically high school. This means a lot of different classes on different days. They won't always have the same class on the same day. For example one day they may have English, Math, Physics, and Geography; then on the next day they may have English, German, Math, Informatic (Computer Sciences), and History. So they are constantly needing to repack their school bag. William has the problem that he really doesn't like packing his school bag, or he forgets to put something he needs in the bag. So he thought it might be an idea to try to build something like this, using what he's learning with the Raspberry Pi. I'm helping him out as he gets stuck, and if there's any programming things he doesn't understand.
That's an excellent idea! As the say necessity is the mother of invention so that works out.
Logged

                 ..... .....

Pages: 1 ... 6 7 [8] 9 10 ... 14   Go Up
Print
 
Jump to: