How can I keep track of my Cloud Resource use?

A judicious usage of one of the cloud "orchestration" tools, such as Terraform or possibly Fugue seems to be the best way.

You can start small, pick a less important and not very extensive environment, carefully decipher it into automatable code and proceed from there.

Broadly this is referred to as infrastructure-as-code, for googling and other buzzword-oriented media applications.

More specifically, say, if you had your environment described in Terraform's HCL files, you would have required ID's exported as "outputs" and could operate on those from scripts/Jenkins jobs/etc


I discovered that it's a know bug of Elasticsearch version (7.10.x), after look this issue and I set to the version 7.9.3 and works fine!


(Rancher employee) Kubernetes itself is really not meant for that use-case.  A cluster is generally a set of machines in the same provider with close proximity (latency) to each other.

Etcd (and therefore API/CLI/UI/scheduling/everything) performance depends heavily on the (worst-case) latency between all members.
Only one cloud-provider integration (storage providers, L4 load balancers) can be configured and most assume nodes are all in the same "region" or similar concept.
Commonly used network plugins assume adjacency and/or provide no encryption or authentication suitable for communicating across untrusted networks/the internet.
Communication between pods/services within the cluster has no consideration for where the "nearest" place to reach a pod for that service is, so latency from a pod to another service can be large (however far apart the two worst nodes are) and unpredictable (sometimes close, sometimes far).  (There is an option for nodePorts to always go to a local pod, but if there is no local pod it will just get dropped).

You can make a "custom" cluster and add nodes from wherever and do what you're asking in Rancher, but you're not going to have a good time (no matter what product is building the cluster).


Running Protractor tests in docker through teamcity

I have been able to figure out how to get everything to work.

Let's start from the beginning:

Changes to config file

Remove the selenium server settings (the docker container starts one on every run, and uses that automatically)
Add 'useAllAngular2AppRoots' otherwise the tests will all fail

The resulting config file will be something like this (ignore the params section, that is just an object with additional parameters you can inject into your tests):

// conf.js
var spec_files = "tests/*.specs.js";
exports.config = {
  framework: 'jasmine',
  specs: [spec_files],
  useAllAngular2AppRoots: true,
  multiCapabilities: [{
    browserName: 'chrome'
  }],
  restartBrowserBetweenTests: true,
  params: {
    sleepTime: 0
  }
}

Setting up the build step in teamcity

Usually when I want to run a build step I do it in a docker container, and execute command line steps inside that container.

To get this to run I could not do this, as we need to inject a few parameters at the end of the run command, so you need to create a docker command, looking like this:


What are reasons to prefer an ingress over using `expose` in k8s?

As @Tensibai indicated:

  With an ingress you can have multiples names on same port, no possible
  by exposing a port tied to only one service


Where do you start with hosting the infrastructure for your infrastructure?

Not a stupid question, but it's lacking details. If by stack you mean Terraform, Vault, Consul etc, then you would go about provisioning machines/VMs/containers for those services likely using some form of ansible or equivalent.

It would be prudent to store those scripts in git or any other source control system.

  Like, for example, should I just be spinning up individual VMs to host each part of the HashiCorp stack? Should I be doing that instead using containers?

Personally, I find containers more flexible especially when upgrading/ testing. The only downside is you have to remember to mount a data volume into the container to persist any configuration data which would normally live beyond the lifecycle of the container.


What are best practices of structuring encrypting variables since Ansible 2.3.0?

Having been working with ansible vault quite a bit recently (specifically regarding what the encrypt and how to encrypt those things without making the code unreadable) I've found very little incentive to change my habits in 2.3.

If I'm going to have a series of encrypted variables, or files, keep them separate (I prefix them all with vault_) and just leave it at that.

A few times I have used the !vault feature but am left distinctly unamazed as it seems easier to just be very explicit about what is and is not coming from a secrets file. That way no one editing my plays makes assumptions about what is and isn't sensitive data.


How to access ephemeral storage of EC2 instance?

You are correct that the describe-instances API doesn't include information about ephemeral volumes. Once an instance is created, the only way to know that it has such volumes is available from the metadata service that is accessible from the instance itself.

According to User Guide for Linux Instances, the mapped ephemeral volumes are shown in instance metadata.

Since you created the instances using Chef, another option would be to check the Chef state database - most probably the information about a created instance includes the ephemeral device information. But this method highly depends on your Chef version and the cookbook you use to create the instances, so YMMV.


This is a known issue with aufs -- see here for context.

Use ADD <tarball_path> and use the overlay driver on the build server.


I have done this in the past using https://github.com/hashicorp/consul-template.

What consul-template does it generate a configuration file (for nginx) based on a certain template you provide. And the values that it fills into this template are coming from the configuration stored in consul.

Each time your micro-services register themselves in consul and need to be addressed in the nginx configuration file, consul-template does it for you by changing the file and sending a SIGHUP (kill -1) to the nginx process that makes it reload nginx configuration.

The repository for consule-template contains an example of managing nginx https://github.com/hashicorp/consul-template/blob/master/examples/nginx.md

If you are using Nginx Plus, which is the commercial version of Nginx. Then it has support for Consul built-in. https://www.nginx.com/blog/service-discovery-with-nginx-plus-and-consul/ and https://www.nginx.com/blog/service-discovery-nginx-plus-srv-records-consul-dns/ go into depth on this subject.


Real-World Validating the Integrity of a Docker Container

2017 expectations

Docker is about getting some freedom from host management at the expense of reinventing all the things package management does. Docker images are just another package format. This is all very new, so I think this question is part of the growing pains of the Docker ecosystem.

basic requirements/criteria

For secure trusted containers, only use them if they have been
 1. tested by current standards
 2. cryptographically signed
 3. checked against a vulnerability database for insecure dependencies

Maybe there is a way, but I don't see a way to un-trust a signed, trusted Docker image after a researcher finds a new vulnerability. This is just one gap in the toolset cited in the question.

Are you planning to use a cloud Docker service? Are the containers' memory encrypted or at least obfuscated at the kernel (containers are just processes)?

Are your Docker hosts secure?

http://ieeexplore.ieee.org/document/7562068/
http://ieeexplore.ieee.org/document/7397244/

economy of security

In the end, you can never escape the economics of checking everything, so there needs to be some respect paid to the speed Docker provides by not worrying about all of this stuff. When you know you are handling sensitive information and you know you need to lock down the execution, it will require measures that are not available yet as commodity solutions (2017). Anything is better than nothing, so get some tools deployed and get your foot in the door.

Hubblestack?

Keep your eye on HubbleStack. It provides a Docker image which can be dropped in to run audits (and remediations) against a live database of vulnerabilities. I'd be surprised if they left Docker trust out of scope forever. Also, if you own the Docker hosts, you can use root to scan all the containers. This kind of thing belongs in the DevOps automated test and validation cycle.

get some kind of checkpoint in place to get security in the pipeline
keep a backlog of unmanaged security concerns (gaps in the tools? CVEs?)
keep the backlog prioritized by what you can get done now, at current funding
get involved in devops development and release planning
deliver software confident that you did the best with the resources you had


Teaching an old dog new tricks: Finally adopting the micro-service architecture

In your approach I see a few scalability problems - I'm judging from a Google App Engine (GAE) context, where scalability is achieved via breakdown of the work in small tasks/work items and strict limitation of the response time for a task execution:

you're duplicating some piece of functionality: both scripts need to parse the same page, leading to your requirement #1
the task execution time for script A is fundamentally unbound, it can vary widely depending on the number of links discovered which need to be checked if already crawled

I'd approach this a bit differently to avoid these issues. I don't even see a need for separate services for this particular job:

the service pulls individual page URLs as separate tasks/work items
it checks if the URL was already crawled and, if so, does nothing, otherwise continues
gets the page content and parses it to extract 2 kinds of info:

the list of other URLs to crawl, creating and enqueueing new work items for each one of them (it doesn't check here if they were already crawled, which may take a long time depending on the number of URLs)
the other info needed from the page

In this approach the duration of each task execution is reduced to the minimum necessary: one page parsing and only one check if a URL has already been crawled. And each page is only parsed once.

The number of pending work items would be the driver for scaling - if it goes above a level new instances can be started, if it drops below a level and multiple instances are running some may be stopped. If implemented as a GAE app the GAE infra can take care of that for you automatically.

This solution can be implemented using a noSQL database (like Google Datastore, in a GAE context), which can scale better than a SQL one.


How to handle large deployments using tools like gitlab and docker

The documentation shows that you can have 40+ branches with different configurations. You express concern that the configuration becomes large. With a good editor that isn't really a problem. Yet that feels wrong, right?

With git, in general, a repository per independent deployable works better than a single repository holding many independent deployables. So an alternative layout is not 1 repo with 40 branches, but 1+40 repos. That would be 1 repo that builds the latest code and creates releases. Then 40 repos that represent the configuration to deploy a particular release of the code configured for one website. The 40 repos that deploy the code can hold the 40 per-website configurations and a standardised build script that simply downloads the right release of the code to deploy along with the website configuration into the environment.

We use 1+N+M git repos. We have 1 repo that holds the deploy scripts. They are versioned and released. We have N repos that build the different apps into releases. Currently, two php apps and two react apps that work together but are deployed and upgraded independently. Then M repos that represent the environments/sites we deploy into which is currently only staging and live.

We are using k8s (OKD) and the deployment scripts are using helm. That just means that our deployment scripts and environment setup are simpler and rollback/downgrade is fully automated. We don't actually download the app to deploy during our build. Instead, we simply push out the docker registry tag of the version of the code we want to deploy into a given environment.

I started a project to opensource our approach to deploying apps from git which is called OCD. It is still a work in progress but it demonstrates some of the above ideas.


Set additional required approver for pull requests at subfolder lever in Azure Devops

Yes you can do this under branch policies.

From the GUI navigate to Repos-> then in the hamburger menu branches-> then select your master branch and then ... on choose Branch Policies. Like so

Then from the branch policies page scroll down and find the Add Automatic Reviewer button and set a path filter based on the folder.


How to use Hashicorp Vault with NodeJS application?

As the Linux philosophy says:

Make each program do one thing well

So it's worth it to use the vault agent, cause it's doing well what it would be done.
You can find an awesome example here


How to pin public root key when downloading an image with docker pull (Docker Content Trust)?


What are the two project layouts offered by Rancher on installation?

Project is a Rancher specific concept. A Project can have multiple Kubernetes namespaces.

The System Project has all the Kubernetes system components like kube-dns, ingress etc. Think of it like System Services running on a machine/laptop.

Default project is where you can launch your workloads to start with. Similar to a scratchpad.

You can create new projects too. For example, Web App, Database, etc. Or it could be based on different users using the cluster. Project-For-Bob, Project-For-John.


How to setup a permanent storage volume for EC2 instance?

That 150GB disk that r5ad.xlarge comes with is known as ephemeral storage - its content is wiped when you stop the instance. If you need persistent storage you can use r5a.xlarge that doesn't have the ephemeral storage disk and simply give it a bigger EBS disk.

Either you can increase the size of the root disk during instance creation or create another EBS and use that. EBS volumes are persistent, they don't go away when you stop the instance (they may be deleted if you terminate the instance though).

If you don't see the previous content of EBS volume after a reboot make sure you're properly mounting it during boot (hint: /etc/fstab).

Hope that helps :)


gitlab kubernernetes integration: configuring the installed runner

Installing kubernetes integration on gitlab
In Admin Area -> Overview -> Runners, the instructions they give are:

Install GitLab Runner
Specify the following URL during the Runner setup: https://<your gitlab hostname>/
Use the following registration token during setup: <insert-token-here>
Start the Runner!

I'm on version 13.13.0 but I don't remember the instructions changing from version 12.x.x.
Project specific runners
As for your question regarding configuration, I'm a little bit confused if you're looking for something beyond those instructions. If you want to set up a runner specific to a project, you can go to a project then navigate to Settings -> CI/CD -> Runners -> Specific Runners. There, there are more instructions on setting up project-specific runners.
Further configuring the runner
EDIT: The executor entirely depends on how the runner itself is installed. If it's installed on a kubernetes cluster, then it must be the kubernetes executor. That will affect your pipeline. I think you can create another one if you install another one with the runner chart. You can also use Helm Tiller.


Reverse Proxy and TCP Optimization Inside the Network

Tuning the parameters you have identified will certainly have an effect - to degrade performance. For example, net.ipv4.tcp_window_scaling is an on or off option. Similarly, net.ipv4.tcp_sack is either on or off. Both default to a value of on on Linux and on all load balancers. Both of these were defined in RFC1323 which was published in order to give us the ability to achieve 10/100 Ethernet networks. Without this RFC, networks were limited to 10 Mbps a second, so disabling these will result in a huge, immediate and noticable performance hit.

These performance benefits come because there is a huge difference between "almost" zero latency and zero latency. Using a sliding TCP window allows equipment to use Selective Acknowledgements (SACKs) to acknowledge the receipt of hundreds of packets simultaneously. So, if latency is 10ms on a network, without SACKs, every packet has to be acknowledged and that 10ms latency adds up very quickly - even at theoretical minimum latencies. This means that I send packet A, then have to wait for a response to packet A before sending packet B. But as it turns out I can send the rest of the alphabet and then some while waiting for the ACK to packet B. Enter SACKs - These allow the receiver to signal that it is has successfully received up to and including packet Z. Meanwhile, the sender is already sending packet AA-ZZ.

This results in something known as a "TCP Window Size" which is the maximum amount of data the client can buffer/handle. Since memory can be a concern, if the recipient is only capable of storing and processing Packets A-F, if we send A-Z some amount of data is going to be lost. Therefore, it becomes necessary for the recipient to signal their ability to receive data - that is "I can only buffer this many packets). This is really CPU dependent on how fast the recipient can empty their buffers and transmit the data up to layer 7. It is not uncommon to see TCP windows scale from empty to full (Max window size goes from 0 indicating the window is completely used to max indicating the window is completely available) and this is one way throughput bottlenecks can be identified on your network.

TCP Window Scaling offers two primary benefits:

Less unused time on the wire - we aren't sitting around waiting for an ACK and we have successfully defeated the prop. delay (latency).
Less time devoted congestion control = more time for data. Congestion control messages can eat into your transmission time and bandwidth. Even with TCP Window Scaling, Congestive Collapse can still occur in which an entire TCP window may need to be retransmitted and congestion control may be required.

Finally, the only real reason to update net.ipv4.tcp_timestamps is to provide additional security as it is otherwise based off of server uptime. Since you are proxying behind a load balancer anway, adjusting this setting is of little value unless you are using something like a Performance Layer 4 profile. I doubt adjusting this will make much difference, though again it is only likely to degrade, not improve performance.

Far and away, what is more likely to help is adjusting your send and receive buffer sizes and using Jumbo Frames. this will allow you to process more data with the same amount of overhead (a 75% reduction in processing over a standard MTU of 1500. These are the primary changes required to reach a 10 Gbps connection. Otherwise, just make sure you have enough CPU to ingest your data from the TCP stack and be aware of the pattern of your traff (eg, avoid things like the C10K problem).

EDIT:

tcp_fin_timeout doesn't actually improve performance, instead it defeats resource constraints. This setting could actually be a major player in a load balancer implementation. The problem is that for any given client and server, the server will be hosting a web server (for example) on 192.168.1.5:80. Every TCP connection must have both a source IP and source port as well as a destination port and destination IP. Source ports are chosen by the client from a pool of "ephemeral" ports. This results in a theoretical maximum of 65534 connections (the maximum number of ports) between a given client to port 80 on that server. When using load balancers, this can result in "port exhaustion" when the pool of ephemeral ports is depleted when SNAT (Secure Network Address Translation) is used on the load balancer, which results in a theoretical maximum of 65534 between the load balancer and the server. There are two basic solutions to this problem: 1) use a SNAT pool where 2 source IPs for the load balancer * 65534 = 131068 connections and 4 IPs X 65534 = 262136 max connections, and so forth or 2) don't use a SNAT. For this to work, the load balancer needs to be part of the default route back to your clients. If you are testing with a single actual client simulating multiple clients, this is a poor real-world test and you may actually support many more clients than you think because you are exhausting your ephemeral ports, where real clients would never do that because they come from unique IPs and two clients can use the same ephemeral port if you are not using SNAT on the load balancer.

Now, I keep mentioning that this is the theoretical maximum but the reality is that on Linux servers, there is a defined range of ephemeral ports and so typically you will see port exhaustion at around half of the theoretical maximum. You can see results similar to adjusting tcp_fin_timeout by adjusting the ephemeral port range in /proc/sys/net/ipv4/ip_local_port_range.

This brings me to the final piece, the TIME_WAIT state governed by tcp_fin_timeout and defaults to 60 seconds. This gives every port used a 60-second "cooling off period" before it is returned to the ephemeral port pool. This is why you saw a performance boost in your testing. This setting is supposed to be 2 times your maximum segment lifetime as defined by the TCP protocol, as it is theoretically possible that for a FIN packet with this setting below below 1 X maximum segment lifetime (30 seconds) to still be in transit. In reality, this is very unlikely, but technically you are breaking RFC compliance by adjusting this setting. If you adjust this setting downward however, you would want to be sure to adjust it downward on the Load Balancer's TCP profile (so tcp_fin_timeout corresponds to Time Wait, not FIN Wait on an F5, for example).

Ultimately, this is a fantastic example of where testing with a load balancer and many servers would result in a huge performance difference - because your testing probably doesn't reflect the real-world architechure. It isn't a good simulation of the customer experience.

However, tweaking both ephemeral port settings and using keepalives are two divergent traffic management strategies. In fact, if you use a setting like OneConnect, there simply is no longer any need to tweak your ephemeral port ranges and timeouts because connections are recycled and multiple clients can be funneled over the same connection (note, this only works for HTTP traffic, but similar can be achieved with tcp_keepalive_probes if you want to configure it that way because clients no longer have to wait for the three-way-handshake - they can simply start transmitting.)

Conclusion

There is no pat answer here. You are asking "Is it worth doing all this work?" - I don't know. Is it? What is the acceptable level of business risk? What does it cost your business if you get it wrong and bottleneck?

Ultimately, I feel you are basically asking two things 1) what do all these settings do and what is the effect of tweaking them? And 2) are there any shortcuts I can take?

Unfortunately, you have not provided enough information to answer either question (which is why your last question was closed and this one will be downvoted.) First, I don't have the time to explain what every TCP setting in proc does (nor do I know off the top of my head, I'm actually googling these on the fly.) but secondly (and more importantly) it is impossible to determine their effect without knowing how your service is architected. There is probably performance that can be squeezed out of your system, but kernel TCP parameters are pretty much tuned to the best possible configuration that can be safely made without breaking the TCP/IP RFC. What you are doing is pushing into territory that can be safe, but only if you understand the architecture and the TCP/IP protocol. And it would behoove you to learn it. If you want to be the smartest guy in the room, learn TCP/IP. Pick up a copy of TCP/IP Illustrated and/or Internetworking with TCP/IP and commit the basics to memory. Learn how HTTP requests and SSL handshakes work. Learn how to read a pcap in wireshark and tell the story of what is happening according to these protocols. This will take you very far in life whether you go in to System Administration, Networking, or Programming (I have used this knowledge in all three realms). Simply put, there is no substitute for knowing what the hell you are doing and this just isn't something you are going to be able to answer off of Stack Exchange. This is why you pay consultants - because if they are good they know the ins-and-outs of this. Once you understand 1) your architecture and 2) what the settings actually do and what effect they will have you will better be able to tell if you can take any shortcuts (that is, I don't need to tweak this setting; it is irrelevant. This testing will be worthless because it doesn't simulate the customer experience.


Docker container is running but when using curl not accessible

I've found solution. Problem is backend took 10 mins to make fully loaded. When curl -m 5 http://0.0.0.0 in validate_server.sh, backend is not ready yet to response back that's why [stderr]curl: (56) Recv failure: Connection reset by peer throw. So, I've added sleep 20s above curl -m 5 http://0.0.0.0 inside validate_server.sh, now it was working.


How to run Ansible file in Jenkins using Ansible plugin

Simple one.
You should either add ansible_ssh_extra_args="-o StrictHostKeyChecking=no" to inventory or disableHostKeyChecking: true into Jenkins pipeline script (inside ansiblePlaybook block).

Like this:

ansiblePlaybook(
  colorized: true,
  installation: 'ansible27'
  inventory: '[hostsfile]'
  playbook: '[playbook]'
  disableHostKeyChecking: true
)


How do I do a dry run with docker-compose?

Welcome to DevOps SE!

A dry run for docker-compose seems to be an open feature request.


Recently I've read a good book about Terraform that I think answers your question (Terraform Up & Running 2nd edition. ISBN-13: 978-1492046905. ISBN-10: 1492046906).

Basically it says that all data in Terraform state file is stored in plain text, including sensitive data. The author says that at the moment of the writing (May 2019) there is no real solution for this, but workarounds like encryption and IAM to control user access based on policies (the book talks about AWS, so you have to find the Azure equivalent and prevent unauthorized access to the state file).

You can find more details on Chapter 3, pages 78 and 79. There is also an open issue at Github about this: https://github.com/hashicorp/terraform/issues/516.


IT staff sizing in the DevOps era - how and when would a DevOps team scale up?

IMHO it's not a good idea to directly and immediately apply a team's gains from its DevOps transformation into scaling the team down. At best you'll lose just the team's motivation to do any such improvement in the future. But most likely you'll lose most/all the talent in that team.

Instead:

allow the team to appreciate the benefits of the difficult transformation they went through, so that they spread the word to your other teams that will have to go through (or are already struggling with) the transformation
focus on delivering better quality products/services, faster - with the same team - this is now possible due to the transformation.
in many cases your business will pick up speed because of the transformation - soon you'll more likely need to hire people - IMHO it's better to use the experience of the ones you already have rather than educating new hires
allow voluntary attrition to work for you and get the effective downscaling down the road.

But generally the DevOps transformations tend to not happen fast enough to allow for the above choices :)

Now onto scaling up/out.

Ideally the DevOps transformation should have greatly increased the predictability of your sw development and delivery process. You should be able to tell with a great degree of precision you existing team's capacity: how much sw/services of a certain quality can your current team get done in a certain amount of time.

When you need more stuff done, or done better/faster than the current team's capacity you need to scale the team up/out. The exact details come from whatever process/methodology you use for your day-to-day activities, probably some agile variant (see also Does my organization need adopt Agile Soft. Dev. before adopting DevOps?)

How - apparently easy: hire more people to handle the overflowing activities. If you're worried about how fast you can get them onboard and actually handling that overflow work at the expected rates: start hiring earlier, based on projections.

The reality can, however, a bit more difficult than that. Scaling a software product/service or the development process and tools supporting it is not necessarily straight-forward, even for a DevOps-empowered organisation.

One of my favourite examples is the weakest DevOps link when it comes to scalability: Continuous Integration. If the CI build fails there can be no Delivery/Deployment (kinda undermining the "continuous" term in the CD context).

Just because a certain team is happily using CI with decent results it doesn't mean the same will happen if the team size doubles. Nope, the results could be devastating - the delivery process can grind down to a halt (in extreme cases). I'm trying to explain this in Congestion in Traditional CI Systems (disclaimer: I'm the founder of the company owning the referenced page and offering a solution to this problem).

Another example could be, for example, the version control system (VCS) being used: as the software being stored into a repository evolves so does its change history, pushing the limits of the VCS system, sometimes beyond acceptable performance levels. Switching to a more scalable VCS system or splitting the codebase into multiple repositories could be an approach to address the problem.

But evolving the development processes and tooling supporting them alongside the evolution of the software products/services being delivered are fundamentally just what a mature DevOps team does. They just need to also be taken into consideration when in the team's scaling up/out strategy.

With that in mind, the how can be, indeed, reduced to just increasing the team size to cover the additional amount of work necessary for execution of the above-mentioned strategy.


How can we automatically transition Epic Jiras based on the transitions of underlying Jiras?

I think this link on the Atlassian Community forum may be what you're trying to do.


How do I automate deployment to Docker with CircleCI?

A few things need to happen for this to work properly. First, add a deployment section to circle.yml:

deployment:
  main: # or whatever your deployment is called
    branch: master # or whatever branch you want to deploy
    commands:
      - docker login -e (your email here) -u (your username here) -p (your password here)
      - docker push pgeiss/appname
      - ./start.sh

Thanks to this blog post for the following script. Then, make a file called start.sh in the top level of your repository (if you use a different name, change the circle.yml's last line) that contains the following:

#!/usr/bin/env bash

echo "stopping running application"
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker stop dodsv'
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker rm dodsv'

echo "pulling latest version of the code"
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker pull pgeiss/appname-webapp:latest'

echo "starting the new version"
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker run -d --restart=always --name dodsv -p 80:5432 pgeiss/appname:latest'

echo "success!"

exit 0

Finally, to allow the script to work you'll need to set the script to be executable with chmod and add the environment variables and your DEPLOY_USER's ssh key (preferably) or credentials (if no ssh key) to CircleCI. After doing that CircleCI should deploy your app after a successful build.


Discovering extra args needed for running ansible playbooks

If an extra variable is truly required, you can add assert statements at the top of the role to stop it from executing if they're not defined:

┌─[jamesph@geror] - [~/temp] - [Sat Jan 13, 09:40]
└─[$]> cat assert.yml
- hosts: localhost
  vars:
    foo: False
  tasks:
    - assert:
        that:
          - "foo is defined"
          - "bar is defined"
┌─[jamesph@geror] - [~/temp] - [Sat Jan 13, 09:40]
└─[$]> ansible-playbook assert.yml
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: Could not match supplied host pattern, ignoring: all

 [WARNING]: provided hosts list is empty, only localhost is available

PLAY [localhost] *************************************************************************************

TASK [Gathering Facts] *******************************************************************************
ok: [localhost]

TASK [assert] ****************************************************************************************
fatal: [localhost]: FAILED! => {
    "assertion": "bar is defined",
    "changed": false,
    "evaluated_to": false
}
    to retry, use: --limit @/Users/jamesph/temp/assert.retry

PLAY RECAP *******************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1


See Azure DevOps WorkItems/Tasks in Planner?

I don't think that this feature is currently available out-of-the-box.  However, you can build a Poower Automation Flow or Logic App that integrates the two of them.
In the long run, I would suggest that you remove Planner from the equation, and just use the kanban board in Azure DevOps for task management. You can display the kanban board inside Teams as well.


Jenkins: Permission issue using Docker as build environment

I had the same issue with node. The thing is files in the container are owned by "root:root". Try adding docker args -u root:root:

docker {
    image 'node:8'
    args '-u root:root'
}


Strategy for deploying/hosting javascript-based static websites in containers

From my standpoint the best approach would be to:

Separate build process using Jenkins that would build NodeJS project into distribution and wrap it into Docker image
Spin up Docker registry that would accumulate Docker images from Jenkins (this registry should be accessible from Kubernetes cluster)
Move environment variables to Jenkins secrets or use separate tool to collect and combine configurations from external Git repo (we use Spring Cloud Config through REST API to collect json/yml definitions for every application in every environment)

Using Jenkins you can configure continuous delivery system based on generic Pipelines. Possible flow would be:

Developers finish their work in correspondence to GitFlow (latest Pull Request merged into Release branch)
Webhook triggers Jenkins pipeline:

Stage 1: collect environment definition
Stage 2: build NodeJS app using npm
Stage 3: build Nginx Docker image with distribution
Stage 4: push Docker image to Docker registry
Stage 5: deploy/update service in Kubernetes using standard definitions

Notifications send in regard of results

This process might be visualized using Rancher. I can answer your questions in chat.


How to keep polling GitHub repository and get commits by it's messages on a regular interval using Jenkins?

Considering you have proper credentials in Jenkins and admin right for Git repo, Create a webhook between Jenkins and your Git repo. Webhook will help you push (reverse poll)the commits. You can use jenkins DSL (groovy) to operate on that Git repo by using Git commands. You can use specific Git commands to list Authors of last commits (or any other requirement) and Use email-ext (or any other) plugin in jenkins to be able to send email through DSL.

One line reference to list Author (yeah..it works as it is in any Git repo)-

git log -1 --pretty=format:'%an'

My reference of email to culprits if something goes wrong my jenkins build

step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: emailextrecipients([[$class: 'CulpritsRecipientProvider'], [$class: 'RequesterRecipientProvider']])])


devops - Optomizing gitlab cicd pipeline's caching

I highly recommend using npm ci instead of npm install. The slow part of installing npm modules is figuring out everything that needs to be installed. If you commit your package-lock.json and use npm ci, it skips that part entirely and just looks at your lockfile to download what's needed. It's significantly faster.


How to monitor AWS S3 cross-region replication process?

Still no answers, and still no CRR monitoring in place on AWS, here's minimalistic solution, any feedback and suggestions are welcome. This monitor will only show which S3 objects are missing from destination, it will not provide visibility on versions of replicated objects.

Prerequisites

Enable inventory reports on src (Source) and dst
(Destination) buckets
Create Athena tables (src_inventory, dst_inventory) based on inventory reports for each bucket

Monitor

Run Athena query after both inventory reports generated e.g. on daily basis, can be executed from Lambda, Step Function, build server or any type of cron job.

Example of Athena query which should show number of missing objects on destination bucket at the time of generation of inventory reports:

FROM src_inventory src
LEFT JOIN dst_inventory dst
ON src.key = dst.key
WHERE dst.key is NULL
AND src.dt = '2018-11-11-08-00';

The result can be reported as CloudWatch metric and monitored with CloudWatch alarm.


What is docker? what problems does it solve?

Docker is a technology that allows you to create OS-level virtualization for software delivery.  There are many, many articles written about what this technology is, how it can help developers and companies, and tutorials on how to get started.

To answer your question of why as a developer you may care (non-exhaustive list):

Building an application that is immediately ready for deployment.
Removing cumbersome installs for quick prototyping and testing.  I use it all the time to setup a SQL server to test scripts before deploying them.
Removes the "It works on my machine" since all the application dependencies are in a "container".
Allows you to easily run and port services that usually would require several minutes of installations and configurations.

For your AWS example, imagine not having to SSH into a server, pull source down, and setup the environment every time you want to scale up or out.  Most cloud providers now have options to deploy your containers and not worry about all those steps (see AWS ECS).

The Docker-Compose you mentioned is slightly different in that it is a way to easily run multiple containerized services together.


Using Code Climate for Test Coverage alongside Jenkins

The issue has been resolved.
This is for nodejs

Need to install necessary dependencies either using yarn or npm.
Once done, run

nyc --reporter=lcov yarn unit unit
$ ./cc-test-reporter after-build -t lcov --exit-code $?


It seems to me that your Dockerfile should be looking like this:

FROM ~~container_on_private_registry~~
USER root
WORKDIR /root
ARG jenkins_jar_file=jenkins.jar
ENV jenkins_jar_file ${jenkins_jar_file}
RUN mkdir -p /root/app
COPY ${jenkins_jar_file} /root/app/app.jar
ENTRYPOINT ["java", "-jar", "/root/app/app.jar"]

Explanation:

We tell the Dockerfile to let Docker know, that it expects a variable named jenkins_jar_file to be passed to it during the build. Subsequent lines can reference that variable with a dollar notation.

Also when you build the image, use --no-cache so it will re-build all the layers of the image.


How to use Vagrant and Docker-for-Windows on a Hyper-V system?

Confirmed by this answer on SO referencing  Microsoft technet once Hyper-V is activated it mask the VT-X instructions of the processor to the OS on top of it and windows run like a guest.

Docker for Windows need hyper-V to work, so disabling it is not really an option.

There's no way to have virtual box running on top of Hyper-V and that's quite normal, you should not run an hypervisor within an hypervisor, they'll fight to orchestrate hardware acces.

The best workaround in my opinion is to create hyper-V virtual machine from vagrant with the hyper-V provider instead of virtual box machines, this way you can create VM or container on the same session.

Side note, vagrant already have a note about this in the hyper-v provider documentation page (found while searching the link to add above):

  Warning: Enabling Hyper-V will cause VirtualBox, VMware, and any other virtualization technology to no longer work. See this blog post for an easy way to create a boot entry to boot Windows without Hyper-V enabled, if there will be times you will need other hypervisors.


macOS testing on Desktop: Docker containers vs. Homebrew installation

One of the advantages of using Docker would be that you can easily run tests with different versions of the (DB) apps, which could be quite difficult with Homebrew (some apps don't easily support multiple versions installed on the same system).


Istio Virtual service - enforce specific api version

I have managed to solve it myself later that day, I had to disable one of the apigroups within the specific crd that is responsible for the given object.
In that case it was editing:
kubectl edit crd virtualservices.networking.istio.io
And then change:

  versions:
  - name: v1alpha3
    served: true
    storage: true
  - name: v1beta1
    served: true
    storage: false

to

  versions:
  - name: v1alpha3
    served: true
    storage: true
  - name: v1beta1
    served: false
    storage: false

Then on redeploy I had the correct apigroup which I wanted.


Chef (well, Test Kitchen) hanging when using local cookbook

It turns out that AWS was just excruciatingly slow and apt-get (applied in the base) was timing out. I would have known this had I gone back earlier and tried testing the base cookbook.


Is containerized deployment of jenkins on ECS better than running jenkins inside EC2 instance to secure pipeline information?

Running Jenkins on EC2 is the better option. It's better to keep the functions that manage your containers separate from the containers themselves. I know this is different than Kubernetes, but this isn't Kubernetes.

Build an ELK Stack on EC2 as well and send your build logs to that instance so you can go through them when troubleshooting. It sounds like your setup is good with data volumes and such. I would recommend that you use Fargate instead of EC2 nodes for your workers though. Not clear which you are using.

There are simpler solutions than Jenkins as well. You may want to consider CircleCI, GitLab Runner, TeamCity, Bamboo, and others. Jenkins is made to do all things and is a good place to learn the process, but if an alternative does the job just as well and more easily, then try that. My current favorite is GitLab Runner. It's just simpler and it integrates your Git repo with the CI tool in addition to having its own image repo for you to use (in certain versions).

You can also give EKS a shot. I haven't tried it but Kubernetes is a superior product. It's far more complex, but definitely a better orchestration tool.


How to manage services inside the container?

Using systemd in Docker is not really that straightforward.

However, according to the doc on the docker hub, systemd is now included in both the centos:7 and centos:latest base containers.
However, you will need to include text similar to the example Dockerfile :

FROM centos:7
ENV container docker
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
VOLUME [ "/sys/fs/cgroup" ]
CMD ["/usr/sbin/init"]

I strongly recommend to read the full description tab documentation on the hub. This will help you make it run correctly.


Why `docker diff` does not see some file changes?

I don't know how I missed that, but the mongo image by default stores data in a Docker managed volume, and not in the container writable layer as I assumed it initially.


Best practices for CI for embedded platforms (ESP32/Arduino)

Since "deployment" to the IoT device, especially in bulk, will not be done using the CI system. Then the purpose of the CI system is mostly to make sure that it will work okay.

This means that you mostly want to run some tests using the CI. Automated testing is definitely possible, both for unit tests and for integration tests.

The most important thing for unit tests is usually very short runtime and a short feedback loop. For a large code base, if possible to compile parts of it using the native compiler of the host and run unit tests there - it is a great option. If you must, you can also cross-compile to the target platform and run the unit tests using some emulator to have the necessary speed. All of this can happen during a CI "build+test" job. Compiling for the target and running unit tests on the target will just be painfully slow if done properly, but can also be done - just make sure to only flash the smallest possible piece of code and have it report back via the Serial that it worked on the device.

For integration tests, if possible to install your code on an actual device and the have some chatter with the device using the serial interface - you can ensure that the code actually works on a target device. If for some reason the code fails or works incorrectly, maybe draws too much power and/or restarts then the integration tests will catch that. You can also have another test device performing measurements and reporting via the Serial interface to the CI job. Can use an Arduino, or another ESP to maybe measure voltage and/or amperage during the execution of your code parts and checking that against known good values. You don't expect the device to pull 400mA when in a deep-sleep mode, for example, this is easy to test with a "test harness".

All of these unit tests and integration tests can be triggered completely automatically on the CI server. Naturally, you will probably have this "server" running on-premises and not rely on a cloud solution that might or might not support your latest integration testing.

With hardware, smoke tests are more "real", since you can actually hook-up the device and see the magic smoke come out. Usually, you don't even need to run code before it burns since the problem is often in the hardware.

Since I mentioned testing, there is also integration testing which is done on the hardware level. If you have a good test harness that checks that the device works well by probing it, you can use the same test harness to do your QC cycle in the factory where they produce the devices later. There is quite a lot of information about making these testbeds. One example for this is here - https://www.sparkfun.com/tutorials/138.


Is there any way of getting code coverage with molecule?

You can use another verifier. Inspec should have a reporter that will report in the correct format. https://molecule.readthedocs.io/en/stable/configuration.html#inspec


The docker login failed.

Error response from daemon: Get https://https/v2/: dial tcp: lookup https: Temporary failure in name resolution

The url is invalid.

change from

docker login -u AWS -p ******** https:/123456790.dkr.ecr.us-east-1.amazonaws.com

to

docker login -u AWS -p ******** 123456790.dkr.ecr.us-east-1.amazonaws.com


This is a similar problem I've been facing. And we're not the only ones it seems. GitLab just added a very interesting feature in their recent 10.5 release where you can pull external files into the pipeline.
In their blog post, they had this to say:

For many of our largest customers, the DevOps team is responsible for providing CI/CD pipelines to a large number of development teams throughout the organization. Previously, this was a painful process to manage. There wasn’t a scalable way to distribute reusable pipeline configuration which meant code needed to be manually copied between multiple .gitlab-ci.yml files in multiple projects. This was a labor-intensive and error-prone process. Additionally, it didn't provide adequate controls to ensure testing and deployment is consistently enforced for each repo.
Starting with 10.5, you can now include external files in CI/CD pipeline definition. Included files can be either local (contained in the same repo) or remote (accessible via HTTP/HTTPS). Including local files allows a large and complex .gitlab-ci.yml to be broken up in to modular chunks that are easier to maintain.

https://about.gitlab.com/2018/02/22/gitlab-10-5-released/#include-external-files-in-cicd-pipeline-definition
This is probably the best way to approach it from a OOP perspective as well: modularize certain aspects of your builds and include it in downstream code.
My peers working at another company do a similar approach with their Jenkins builds. They've essentially build on a 'makefile' for their builds that is based on templates and evaluates certain variables set in the Jenkins UI. Based on those variables, the code will then fetch (curl or git) 'module' files from a central 'builds' repository. It then uses all the modules to compile a single pipeline bash script and runs it.
This allows central changes to be done to a single file, that all the builds inherit.


Docker is going to be far more well-known than Mock. I consistently see Docker used during build in companies that I work for. I think I have seen Mock before, but only because I am a Fedora user.

However, Docker is not a build tool in of itself, so you can still use Mock inside a container. This is a great way to avoid the "snowflake" build server. By using Docker containers, your builds are consistent no matter what build tool you are using. You can standardise on Docker as a platform for builds, and leave developers to choose the best build tool (whether that's Mock or NPM).

The only other suggestion I'd have is to run the builds on a CI server, and Docker makes this really easy.

Here's one example of a company using Docker for builds (as well as tests and deploys).


Kubernetes: How to expose a Pod/Deployment to the Nodes Network

Some clarification first,
the ClusterIP is only accessible inside the cluster this is not the right choice to solve your problem and the LoadBalancer depends on the integration with your network, real or virtual, to generate the correct resources and assign an IP address.

With that said, an easy way open a service port should be to use services of type NodePort it assigns a port directly on the machine, the same way you do when you open ports on a Linux machine, but with the difference that all the process is managed by Kubernetes. In that scenario, you only need to worry about the routes between the networks managed on your router.

You have more info about NodePort on the official Kubernetes documentation here

UPDATED

As per the last comments, for the use case, the use of NodePort should work and probably is a good option to use some DNS management like external-dns to not worry about how to setup the right IP.


What exactly are "Truly reproducible builds"?

What exactly are they?

Here is a quote from reproducible-builds.org:

  Reproducible builds are a set of software development practices that create a verifiable path from human readable source code to the binary code used by computers.

Why are they important?

IMO the easiest way to explain their importance is to consider them as a variation of a backup procedure.

As an example:

Assume a business which uses (depends) on some software package licensed from some software vendor. Whereas the business only gets the executables, not the sources, etc that were used to create those executables.
Everything goes well, but at some point something goes wrong with the software vendor, e.g. they go out of business (eg bankrupcy).
This may expose a risk to the business (in the long run). I.e. if there is no procedure/agreement in place for the business to get (legal) access to all required sources, documentation, build procedures, etc related to anything from the software vendor used (back in the days) when the executables (used by the business) were created (and shipped to the business).
That's were "Software Escrow" comes to the rescue: if there is such agreement in place, one would think that via a 3rd party, it would still be possible for the business to get access to "whatever was used" to be able to reproduce the executables, so that from there on, the business might have a chance to continue to use that software, and where apropriate start maintaining it themselves (for only running their own business).
However, the "whatever was used" in the previous bullet is the toughest part to make this work. It requires that the 3rd party performs appropriate validations upfront. And trust me, it takes a while before you can recreate an executable for which you can prove that, apart from (eg) the link date, it is a perfect match with what the software vendor delivers to the software agent.

And why are they so difficult to create?

If the above sample is still not clear enough, imagine you're my software escrow agent, tell me what you need as input to recreate a copy of the software licensed by my customer. Get it? You didn't forget to check about which version of my compiler, maybe my OS, compile/link options, versions of reusable components (includes), libraries, etc?


How to Allow AWS S3 Access From only Specific IPs

please attach below policy to your AWS S3 Bucket.

  Allow Access to Specific IP Addresses

 <div class="code">
 {
   "Id": "S3PolicyId1",
   "Statement": [
     {
       "Sid": "IPDeny",
       "Effect": "Deny",
       "Principal": {
         "AWS": "*"
       },
       "Action": "s3:*",
       "Resource": "arn:aws:s3:::bucket/*",
       "Condition": {
         "IpAddress": {
           "aws:SourceIp": "54.240.143.188/32"
         }
       }
     }
   ]
 }
 </div>

  Deny Access to Specific IP Addresses

{
  "Version": "2012-10-17",
  "Id": "S3PolicyId1",
  "Statement": [
   {
    "Sid": "IPAllow",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": "arn:aws:s3:::bucket/*",
    "Condition": {
      "NotIpAddress": {"aws:SourceIp": "54.240.143.188/32"}
    }
   }
  ]
 }


Using HTTPs in application vs reverse proxy

You will find people from both camps (terminating TLS as soon as possible vs terminating TLS in the application code) but it's not as serious as for example tabs vs spaces.

The benefit of terminating TLS by reverse proxy is that application developer does not have to implement TLS in the application itself - possibly introducing bugs. Another benefit mentioned by jdog is the CPU cost; which is negligible these days - unless you are running 100s of micro-services and it all adds up to a large chunk of money.

Personally I always go with reverse proxy terminating the TLS unless there is a specific requirement against it (security reasons). I also make sure that service behind reverse proxy listens on the localhost / local network only - is not accessible directly form the internet.

You may find wikipedia article about TLS termination useful.


what environment does a build "RUN" run in?

In the Dockerfile, the RUN step is performed during the docker build process. The operating system is shared with the host (containers != VMs). The filesystem and rest of the configuration used to define the temporary container is based on the state of the image resulting from the previous step. Each step in the Dockerfile extends the image with additional metadata and filesystem layers.

The RUN step itself is performed inside a temporary container. The output of the RUN step is any filesystem changes that exist in the container at the time the command exits successfully. Changes to the shell (like a cd command or setting an environment variable inside the shell), spawned processes, and changes to files inside a Volume are all discarded. The changes to the container filesystem are captured as a filesystem layer that is included in the resulting image.


What is Titus and what issues does it solve?

It's different to Kubernetes in that it seeks to leverage existing AWS services instead of bolting on top. Kubernetes supports a number of cloud providers and contains a number of components that exist in AWS already.

Although we're starting to see better kubernetes AWS integration and with AWS joining the cloud native foundation this is only going to get stronger.

Titus seems to be an alternative to Kubernetes but its not available yet. My bet is AWS will strive to better integrate K8 and likely Titus won't take off even when it is released.


call container B inside container A using docker networks

If you have a user-defined network set to bridge, you can access other containers by their container's name which is automatically as hostname for containers inside same network.

So from containerA you could do ping containerB without problems. In your case it would be changing the setting to containerB_name:(container B needed port)/route


Gather resource usage by process in a kubernetes cluster

There are several alternatives to Prometheus for monitoring your cluster.

While some like Heapster have been superseded by Prometheus, other solutions such as cAdvisor may have the functionality you are looking for.


How to integrate multiple "feature" react apps into monolithic PHP repo?

What approaches are there to manage multiple package.json files that live in each feature directory

One could leave the package.json in every feature directory and let the CI read the package.json when building the app. One could define the versions of dependencies in the package.json to get control about the app. If the app works with version A.B.C of a certain library then one could define this version to prevent that an update will break the app.

  How do we identify common packages and lock them into a root /package.json file. (Independently, all developers already use yarn)

Discuss this with the team and figure out what is the best solution for everybody. Multiple solutions are possible. Everybody should be happy with a certain approach.

  How to accomplish this with as little friction as possible for all react based developers - so that they get to simply continue with
  whatever they are already doing and DevOps handles the gritty bits?

DevOps in my opinion means that a certain team that consists of multiple specialties, e.g. test, dev, ops is responsible for a certain app. This means that the whole team should agree with a certain approach and everybody should work and handle issues according that process instead of DevOps handles the gritty bits which is about silos and "throw it over the wall"


Why container can use ifconfig to get the host information?

You are not accessing the hosts network interface information. Rather you are receiving data of the bridge adapter created for your Docker Containers Network.

Please note the results from my web server:

Container networking info:

# docker exec -it ghost sh
/var/lib/ghost # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:12:00:02
          inet addr:172.18.0.2  Bcast:172.18.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:20 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1536 (1.5 KiB)  TX bytes:0 (0.0 B)

Host network info:

root@li293-xxx:/data/mydomain.us# ifconfig
br-08d99d9f172e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
    inet6 fe80::42:57ff:fe74:c49b  prefixlen 64  scopeid 0x20<link>
    ether 02:42:57:74:c4:9b  txqueuelen 0  (Ethernet)
    RX packets 150  bytes 16535 (16.5 KB)
    RX errors 0  dropped 0  overruns 0  frame 0
    TX packets 234  bytes 21425 (21.4 KB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
    inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
    ether 02:42:82:3d:17:63  txqueuelen 0  (Ethernet)
    RX packets 0  bytes 0 (0.0 B)
    RX errors 0  dropped 0  overruns 0  frame 0
    TX packets 0  bytes 0 (0.0 B)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 66.xxx.xxx.205  netmask 255.255.255.0  broadcast 66.xxx.xxx.255

Please review the networking documentation at https://docs.docker.com/v17.09/engine/userguide/networking/


lc_collate values for database "postgres" do not match: old "C.UTF-8", new "en_US.UTF-8"

I solved this thanks to Reddit post. It was fairly simple when combined with another part of my testing.

I used the code {PGBINNEW}/initdb --pgdata="$PGDATANEW" --encoding=UTF8 --no-locale to reinit the DB AFTER I did the below command:

sudo rm -rf ${PGBASEDIR}/data10

What happened was I cleared the old DB init config and then put in the config I needed without the localization of the database


How to setup artifact filters to differentiate for deployment

Azure DevOps supports artifact filters for each environment. Below is screenshot of how to enable the feature.


Is there a premium version of Google translate that is available for purchase to integrate into business applications?

Yes !

There's an API available on Google Cloud & you can find more details about it on https://cloud.google.com/translate/


error: exportArchive: The operation couldn’t be completed. (IDEDistributionErrorDomain error 3.)

https://forums.developer.apple.com/thread/75038

  Hi!

  I think I've got it figured out...

  instead of:

-exportProvisioningProfile "MyProvisioningProfile"

  use:

PROVISIONING_PROFILE_SPECIFIER="MyProvisioningProfile"

  We've got unity building the project every time and it doesn't fill in
  the PROVISIONING_PROFILE or PROVISIONING_PROFILE_SPECIFIER. And it
  looks like it will figure out the alphabet soup that belongs to
  PROVISIONING_PROFILE by looking it up itself provided you have
  downloaded it already.


How can I access directories inside a docker container from outside (not the reverse!)?

I think you may be using the wrong tool for the job.  Docker is not designed to be a version control system, it is designed for containerization.  Furthermore, your question makes it sound like the development and support team members are constantly changing the files that are essential to the makeup of the image.  This makes using a volume pointless as the Docker image will always be changing.

If I'm understanding what you are trying to achieve (people should be able to modify the src and eggs folders with whatever editor they want), then you may want to explore adopting a CI/CD workflow:

Committed changes to the src and egg folders trigger a new image to be built.
The new image is pushed to a Docker repository.
Team members can then pull the updated image.

As for your two specific questions:

  Do we usually put build stuff that are actually part of the container into volumes? The container has no meaning without these volumes.

Not usually.  Generally if I have to make changes to a container I would remote into it and then commit the changes.

  Can end users take a copy of these volumes and just modify it according to their convenience without the master copy of these volumes? (Like Image vs Container concept?)

This is sounding like a version control system which Docker is not designed for.  You can have a shared volume that users have access to, but the use case is for persisting data outside of containers (e.g. SQL database data).


cannot import OpenCV in a Docker container in Raspberry Pi

Seems to be just an issue with a Python package.

Traceback (most recent call last):
  File "a.py", line 1, in <module>
    import cv2
ImportError: No module named 'cv2'

Is a common Python error, basically it can't find the package cv2.

You could try and install OpenCV through pip instead of installing through apt get.
Something could be up with dependancies that you may have missed.

RUN pip3 install opencv-python


Error 502 before execute Ansible playbook

First - @caxcaxcoatl is completly right - your playbook is not Ansible, it is a bash script inside Ansible and that isn't Ansible.
Your error doesn't have something to do with your playbook, but with that, what the playbook is doing. You're redeploying an application and then the proxy in front of the Tomcat cannot read the response from your Tomcat. So your deployment is wrong, has some failures, what ever. You should have a look at your Tomcat HTTP response (without the Proxy in front) direct on the machine (with curl, telnet or whatever). Maybe you should also check the logs of Tomcat to get the error. The proxy sends the error, because the Tomcat is answering with a 500 HTTP error code, when the proxy hits /portal-client/cm/misalud/.
To give you a hint, how a playbook could look like (of course this wasn't the question, and it could be done much better then this with includes for a unspecific WAR and just call include_tasks on all available WARs in the main playbook, or with a Ansible module written in Python just giving the params of the new WAR etc.):
- name: Deploy war
  hosts: grupo1
  vars:
    webapps_cur "/opt/tomcat/catalina_base/webapps"
    webapps_old: "/opt/tomcat/catalina_base/webapps.old"
  tasks:
    - name: "Stop Tomcat"
      systemd:
        name: tomcat
        state: stopped

    - name: "Delete old Backup WARs"
      file:
        path: "{{ webapps_old }}/{{ item }}"
        state: absent
      with_items:
        - "portal-admin"
        - "portal-client"

    - name: "Backup the current WARs"
      copy:
        src: "{{ webapps_cur }}/{{ item }}"
        dest: "{{ webapps_old }}/{{ item }}"
        remote_src: true
      with_items:
        - "portal-admin"
        - "portal-client"

    - name: "Delete the current WARs"
      file:
        path: "{{ webapps_cur }}/{{ item }}"
        state: absent
      with_items:
        - "portal-admin"
        - "portal-client"

    - name: "Deploy new WARs"
      unarchive:
        src: "/tmp/{{ item }}.war"
        dest: "{{ webapps_cur }}/{{ item }}"
        user: "tomcat"
        group: "tomcat"
      with_items:
        - "portal-admin"
        - "portal-client"

    - name: "Start Tomcat"
      systemd:
        name: tomcat
        state: started


Since the timeout is a Helm option and not a Kubernetes object attribute, it can't be embedded in the chart itself.
The --timeout option is the amount of time the Helm utility will wait for Kubernetes commands to complete before marking the release as FAILED.
More Info on Helm CLI options


Code coverage versus functional coverage

The way I would approach this would be to have a second part of the junit test that reads the generated zip file and then verifies it is correct. You are right that if you are only ensuring you can generate the file, but the file is incorrect, the unit test is incomplete.


Automate Helm's tiller installation with Terraform

How I see there is two options (from my exp):

Use configuration for provider.
https://www.terraform.io/docs/providers/helm/index.html, there is option to install tiller.
Use prefabricated modules, for example:
https://github.com/sagikazarmark/terraform-tiller
https://registry.terraform.io/modules/Vidimensional/tiller/kubernetes/0.0.1


What are best practices for setting SLOs for a new service?

In an ideal world, the SLOs would be driven by business needs, and information about your customers.

Remember: an SLO should be the level of reliability at which most users are happy with your service.

To do this you can partition your service's behaviour into critical actions.

A critical action is a discrete piece of functionality offered by your service, and which your customers care about. It is typically defined from the point of view of the customer, and independent of the implementation (eg., "Customer successfully added an item to their shopping cart", "Customer viewed the high score table")

With some critical actions chosen, decide what the correct SLI specification should be for each action: does it succeed or fail, and needs an availability SLI; does it take some time to complete, and need a latency SLI; does it produce data to be consumed later, and needs a freshness or a correctness SLI? And so on. (Adding an item to a shopping cart should have an availability and a latency SLI; viewing a high score table should also have freshness SLIs, and potentially correctness)

With these specifications you can start to think about what the impact of unreliability would be on your users: will one failure every few days make a user unhappy? What about one failure per hour? Are failures masked by client-side retries? Will users be annoyed if clicking a button results in a 100ms delay before something happens? What about 1000ms or 10 seconds?

At this point, you have enough information to start setting SLOs and thinking about how to implement your SLI specifications: making trade-offs between fidelity, cost, and complexity. It is here that you can take advantage of the fact that the service is still being designed, and ensure it is designed with observability in mind. For example:

Splitting the service into microservices such that those microservices map neatly on to different SLIs and critical actions
Ensuring that any HTTP requests that should be counted towards different SLIs can be disambiguated at the load balancer (eg., don't have all requests be POST requests to /handler)
Include logs and metrics (eg., via OpenCensus) in the design from the start and consider their impact on your application's scalability and reliability.
If you plan to use synthetic probing, allow some way to do this (consider how much of a web page's logic is in client-side scripts, as opposed to server side, and how you would monitor this)
Can you include client-side instrumentation, and how would you collect the results?

The most important thing when setting an SLO for a new service is to plan to iterate: Even if you don't know what level of service is necessary to make users happy to begin with, you should have some metrics you can use to determine this (satisfaction scores, ticket levels, random surveys) and use these to tighten or relax your current SLO as necessary.

Don't hold your service to an SLO that is harder to defend than your customers require, but do ensure that your SLO is sufficient to keep them happy.


No Kubernetes Option in Docker Desktop for Windows

You need to first switch to Edge version by going to the

Docker -> Settings -> General tab.

Then, you have to enable the kubernetes.

  Open Docker for Windows settings by accessing your system tray and navigate to the Kubernetes tab. Select the checkbox for Enable Kubernetes to start a single-node cluster when starting Docker.

Reference: https://medium.com/@damiannolan/kubernetes-on-docker-for-windows-5ca0c6395668

As per this old link and a latest one, Kubernetes needs a Linux master node and thus the kubernetes cluster can be initialized only from linux containers in Docker desktop for Windows. After enabling and starting the docker containers, you can add Windows containers the the running kubernetes cluster.

This will give you much clear context.


What are best practices and/or tools for updating AMIs on a regular basis?

We are planning to automate this process for our Jenkins EC2 slaves.

Currently we have to manually update the AMI ID every time we build a new AMI, but by taking advantage of the config.xml file where Jenkins store all its configuration, we should be able to automatically update the AMI value in this file and then restart Jenkins to take those changes into effect.

EDIT: see comments below for a better way to make those changes rather the config.xml file


How to connect to a kubernetes cluster that is inside of a docker container?

While you are running a container linked to your PC/Server Docker, the Kind Kubernetes port is available on your PC/Server not on the container.
So if you run kubectl commands from your machine it will work (including the kubeconfig).
Solution: Using --network="host" in your docker command, the container will act as your localhost (eg. docker run --network="host" -v /var/run/docker.sock:/var/run/docker.sock -it ef33f1a0ac5e bash


When would I choose IPTables over an Ambassador pattern for port forwarding?

iptables will probably scale better than socat because socat runs completely in user space.  There's a common impression that iptables is hard to deal with, but I find it easier that most routers to fiddle with.  There are a number of iptables wrappers available to make it easier to handle.

If you're willing to accept the performance hit of running in user space then consider haproxy over socat because it is intended to run over longer periods and provides statistics, failover, SSL termination, and lots of other conveniences.


Troubles with mapping ports on a very simple Docker container

May be you are using bridge network mode. In that case, you need to bind your host port to container port to use same port.
So when you execute docker run, you may need to add
-p 5000:5000 as a flag to docker run command. Also I don't see any expose in Dockerfile. So you may want to add EXPOSE 5000 in Dockerfile just above cmd.


Error while following Docker get-started part 3

So what's the problem?

  'app.py': [Errno 2] No such file or directory

I think you already answered your question. The troubleshooting part, you suspect something, but you did not test it to see if it is really what you think.

Create two more files, requirements.txt and app.py, and put them in the same folder with the Dockerfile. This completes our app, which as you can see is quite simple. When the above Dockerfile is built into an image, app.py and requirements.txt is present because of that Dockerfile’s COPY command, and the output from app.py is accessible over HTTP thanks to the EXPOSE command.
Get Started, Part 2: Containers


Historically Octopus Deploy was a solution to deploy software on Windows only, therefore the server component only supports installation on Windows.

This is for a couple of technical reasons:

Octopus Server takes a hard dependency on Internet Information Services (IIS) which itself only runs on Windows.
Octopus Server is written in the .NET Framework doesn't run on Linux.

Once you have the Octopus Deploy Server software installed, you can manage Linux hosts via SSH, you don't need tentacles to manage Linux hosts.


Alternating VC branches to build deployment artifacts for different environments

You can create a release process to manage deployments in your environments.
A plugin that can help you is the maven-release-plugin, where you 'freeze' the package and can promote them between environments (DEV -> UAT -> PROD). This ensures that the artifact (package) is exactly the same as the Production environment.


Automated Integration Testing of an API in Azure Functions using VSTS

There is no difference on vsts for your test (unit, integrated or functional). You just set the right dll name with your tests:

or select Test plan if you have a test plan with associated test recordings


How to render assets when building a static website with GitLab Pages and Jekyll

It appears the issue is in the URL used to access the documents in the blog post excerpt. First of all, the URL should begin with a / since the assets folder is at the root of the site project. However (and most importantly), I also had to add the baseurl to the file URL as shown below:
blog post excerpt:

    {% for duck in ducks %}

{{ duck.name }}

View | Download

    {% endfor %}

If I weren't building my post entries from a template the link would actually be /{{ site.baseurl }}/assets/v{{ version }}/{{ duck.url }}, but Jinja does not understand the site.baseurl object when rendering the template. To bypass this, I read the value of the predefined GitLab CI_PROJECT_PATH variable and pass it to the template under the name baseurl when rendering.


I managed to setup an environment that does what I described above. Here are the full details:

Launch a template that creates a CodeCommit repository for the
application. This repository will be used to store both the source
code and the CloudFormation (CF) templates to create the required
infrastructure for the application;
Launch a template that creates a CodePipeline pipeline for the
application. The pipeline will deploy a stack for the infrastructure
and one for deploying the code through CodeDeploy (the CodeDeploy
stack will will create a new CodeDeploy Application and deployment
group, and will deploy the latest code to the infrastructure created
previously)

The pipeline has a paramater "StackBlueeGreen" that is used to determine whether the stack is the Green or the Blue version.

When the pipeline runs successfully, it will create a separate infrastructure

  (NameOfTheApplication-B or NameOfTheApplication-G)

and deploy the code there. The second infrastructure will have its own load balancer listener rule setup with a staging URL

  (something like URLOfTheApplication-staging.somedomain.co.uk)

This allows developers to test the new version of the application withuot affecting the previous version. Once developers are happy with the new version, the listener rules for the old and new infrastructure need to be changed manually so that users are now directed to the new infrastructure. If there is an issue with the new version, rolling back is as simple as swapping the load balancer listener rules again.

Finally, the pipeline stack need to be updated: the StackBlueGreen parameter needs to be switched to G/B depending on how it is currently set. If this is not done, next time there is a change, the new version will be deployed on the same infrastructure.

[The setup described above works very well and meets all the requirements outlined in the OP.
An improvement that I would really like to implement is automating the switching of the load balancer listener rule, deletion of the old infrastructure and updating the pipeline stack. I will add to this question if I manage to implement this improvement.
Meanwhile, any input would be much appreciated!]


Azure DevOps Pipeline Conditional Flow with Manual Intervention

@Shayki's answer was a large portion of what I was looking for (thanks!), but I was eventually able to figure out a solution that met all of my requirements. It involves a less than ideal use of the Release Description to propagate state between jobs.

The updated flow diagram looks like this:

I was originally thinking of the conditional flow as an if else statement, but it is really more of multiple if statements where the condition is the same. If flag is set, do X; if flag is set, do Y; etc.

Details

Instead of running the agentless job when a previous condition fails (which would result in the entire deployment failing, a violation of one of my requirements), I created a custom condition that the agentless job runs as:

If all the previous tasks have succeeded, and if the Release.ReleaseDescription variable contains the string defined in the _dbDeployErrors variable, then this job will execute (and the manual intervention task inside of it).

The _dbDeployErrors variable isn't too important, I set it to ;DB_DEPLOY_ERRORS_$EnvironmentName. The main idea is that, at least with the Web Designer, there is no way to pass information from an agent job to an agentless job. So I wrote a powershell script which uses the Get Release and Update Release Azure DevOps REST API to update the Release Description parameter on the release. This allows me to pass information to the next job in the deployment.

The basic steps for setting this flag are:

Run the db deployment
If the deployment has errors, do not fail task, but instead set an agent variable
Next task determines if the variable is set, and if so calls the Update Release API to append the flag to the release description
Future jobs and tasks can now analyze the state of the release description variable

Note: I set an agent variable instead of making the API call from the db-deploy script so that I can keep my deploy script clean and maintainable. I move the responsibility of calling the API to a dedicated script instead.

If the Manual Intervention is resumed, then the third job executes as an agent job. It will attempt to deploy the db again, but this time it sets a -Force flag on the script to ignore possible data loss (it was approved by the one who performed the intervention). If this deployment succeeds, then a quick cleanup is run to call the Update Release API and clear the flag from the release description. Finally, the other tasks can resume as normal.

Here is what the tasks in my final release definition looks like:


Are there any tools to monitor custom console applications on Windows Server?

This is probably a question better suited for ServerFault or StackOverflow.

One approach that may work is to Create a Windows Service from an Executable, then set the service to restart on failure/crash.


Jenkins X is a CI / CD platform for Kubernetes.
This project is a sub-project of Jenkins.
see below
here


jenkins pipeline code to build docker images

On building the docker image, you need to have the docker daemon enabled.
You can also use the jib, a Google project, which creates images for your Java applications without the Docker daemon


GitLab: Make commit in (shared) CI runner and push to repository/open merge request

CI token used for code checkout is not allowed to do writes to the git repository (unlike GitHub Actions). You need to generate a new personal token and add it to secrets.
See our integration project here as a sample: https://gitlab.com/taleodor/sample-helm-cd
Note, if you search through StackOverflow, you may find several other ways to achieve this but in any case you need to explicitly pass valid credentials that are able to write to the repo.


Is it possible to downgrade Jenkins container plugin thought CLI?

Check out Config as Code for installing plugins.

If you're using the jenkins image, just use an older version of the image that has the proper plugin version


How to list all users with specific role in GCP

That can be achieved using another gcloud command:

gcloud beta asset search-all-iam-policies --query policy:"roles/owner" --project $your_project_id --flatten="results[].policy[]" --format="csv(bindings.members[0])"


docker micro-service cannot connect to bitcoin docker

the containers hosts file will differ from your hosts hosts file.

see it kind of like a VM.

Also 127.0.0.1 is the wrong address, you might need to configure the actual ip/hostname your host has and where the service is listening on inside the watchblockchain container.

So Problems:

Your docker container knows nothing about the host bitcoin. an entry in your docker hosts /etc/hosts won't change that.
Maybe your bitcoin service only binds to localhost 127.0.0.1 - this will also keep any docker container from connecting to it. Maybe you can configure it so that it binds to the docker bridge IP. So only all the docker containers are able to connect to it.
It would be easier if your bitcoind would also be a docker container and was configured inside the docker-compose.yml. That way you could resolve the host using the container name. But still your bitcoind should be configured to bind on every interface.

Hope this helps ;)


Is there a 5$-30$ IaaS that supports some rolling-release distro?

Linode is one of these (see linode.com). Linode offers 5$ ArchLinux and yesterday I talked with their staff and two staff members confirmed to me that they can't find any plans to stop supporting it in the future, so that's good.

According to ArchLinux wiki here, Arch Linux is a full rolling release OS which is basically "true rolling release OS":

  Linux Mint Debian Edition (LMDE) is based on Debian Stable and only
  receives updates in Mint packages and security updates. Arch is
  instead a full rolling-release distribution.

I understand that to be the highest one can go in rolling upgrades and of release as well.


How to set Terraform Cloud environment variables from the command line?

You can use the Terraform Cloud provider to manage workspace configuration in Terraform Cloud in the same way as you might manage any other infrastructure objects with Terraform. For historical reasons the provider is named tfe, but it can be used to manage both Terraform Enterprise and Terraform Cloud.

In particular, you can use the tfe_variable resource type to manage both Terraform variables and environment variables for a workspace:

variable "environment_variables" {
  type = map(string)
}

variable "workspace_id" {
  type = string
}

provider "tfe" {
  hostname = "app.terraform.io" # Terraform Cloud
}

resource "tfe_variable" "test" {
  for_each = var.environment_variables

  key          = each.key
  value        = each.value
  category     = "env"
  workspace_id = var.workspace_id
}

To integrate this with the script you've already written, you could perhaps use PowerShell's JSON encoder to produce a file azure.tfvars.json with content like this:

{
  "workspace_id": "organization/workspace",
  "environment_variables": {
    "ARM_CLIENT_ID": "...",
    "ARM_CLIENT_SECRET": "...",
    "ARM_SUBSCRIPTION_ID": "...",
    "ARM_TENANT_ID": "..."
  }
}

You can then run the above Terraform configuration using a command like this:

terraform apply -var-file=azure.tfvars.json

When considering automation of this sort it's worth considering where the Terraform state snapshots for this "early" Terraform configuration would live. You could potentially create an additional Terraform Cloud workspace for this special configuration and be sure to configure it to not use remote operations, so that a local run of this one workspace can then arrange for your other workspace to support remote operations.


How to create unique GPG Keys for different environments / projects

If you want to create separate key pairs that are not dependent on your personal keys, the easiest way is to use a separate GPG home directory for each. For example, to create keys for "projectB", first create an empty directory, then tell gpg to use that dir as its home:

mkdir projectB-gpg
gpg --homedir ./projectB-gpg --gen-key

That will do what you want, although you will now have the problem of distributing the private keys and passwords securely to each environment.


How can I assign an IP address to a docker container different from the host's IP?

SOLUTION APPROACH

The approach is to create a macvlan. This will create a virtual adapter, that is allowed to lease an IP address from the subnet defined.

1) Activate Promiscous Mode

For the virtual adapter one must enable promiscuous mode in the network. For Unifi controllers, do a SSH to your gateway and set:

ifconfig [interface] promisc

Some network gateways/controllers have set promiscuous mode by default. Continue with step 2 before you're wasting time to find that out.

2) Create macvlan

Then one can create the macvlan.

$ docker network create -d macvlan \
  --subnet=192.168.2.0/23 \
  --ip-range=192.168.2.5/25 \
  --gateway=192.168.2.1 \
  --aux-address="my-router=192.168.2.10" \
  -o parent=eth0 unifinet

Limiting the IP range: use --ip-range to scope the possible IPs to lease.
Avoiding certain IPs: The --aux-address marks my Synology host, which should never ever battle for that IP address (It's marked in the unifi controller's DHCP service as a fixed IP anyways).
the -o parent is your network interface you want to attach your macvlan. In my case unifinet.

Check with docker network ls if the macvlan has been properly created.

NETWORK ID          NAME                DRIVER              SCOPE
c49094a4c914        bridge              bridge              local
b2315de1aa7e        host                host                local
c124eda0f9d2        none                null                local
a60da50f0d12        unifinet            macvlan             local

3) Start Docker Container

While you define the image (--name UnifiController jacobalberty/unifi) assign the freshly created macvlan (--network unifinet) to the container (UnifiController). It will grab the latest image respectively the latest Unifi Controller version available from Docker Hub.

docker run -dit --network unifinet --name UnifiController jacobalberty/unifi

Verify if the container has been assigned to the network in the container section of the JSON config.

sudo docker network inspect unifinet.

[
    {
        "Name": "unifinet",
        "Id": "a60da50f0d1229d1a3c76210141e0c81567c17daf6c2b49d4f1c83d5ec9f02b3",
        "Created": "2019-10-04T14:39:37.377311991+02:00",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.2.0/24",
                    "Gateway": "192.168.2.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "b819bf1d6f3af459825a1a7b58f9a44e15e6e09489e7bf50653ed8e1e176fd73": {
                "Name": "UnifiController",
                "EndpointID": "41a048034ad63e48b46b58aed65661c9eaa2bf6937d3eebacefde4478ad26cce",
                "MacAddress": "02:42:c0:a8:02:02",
                "IPv4Address": "192.168.2.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "parent": "eth0"
        },
        "Labels": {}
    }
]

The IP 192.168.2.2 is assigned. The container workes properly and stable.

4) Optional/Reminder

If the IP was not assigned respectively one cannot connect to it for any service, the follow up on the promiscious mode setting as described in Step 1.


How to change existing policies in an organisation?

You have to understand that processes change people who follow them. As people learn, internalize and get better at a process, it changes the way they learn how to solve a particular problem. A set of similar processes reinforce each other into a mindset the person uses to solve a category of problems and eventually form a set of values that guides decisions and new solutions to new problems.

Even if you change the process, without the change to the mindset and even more crucial to the values, the person will simply adapt the new process to conform to the same values, same mindset or even same solution as in the original process. At a certain point, it is not possible to divorce this person in this position from the mindset acquired or change the underlying values.

To institute a change, you have the following two options:

Bring in a person who will already have the right values and mindset and in best case scenario, understand the process that needs to be followed without your help.
Bring in and empower new employee, either recently hired, a new hire or a transfer from a different team in the organization and train him in the new process, hoping to instill the new mindset, hoping the new set of values will emerge.

If the change is local, you might prefer an internal transfer as that person would already share the global company wide values which you want to retain. In case of a larger change, you need to bring in someone from outside to have a fresh perspective and to not share the company wide values which you might be trying to change.

The important part is to empower the person, team or business unit to follow the processes and insulate them from the old team, other teams or the rest of the company, respectively, which might still follow the old set of processes. As it is very hard to insulate such change agent from management above, if the change is to be larger, it often needs to follow all the way up the management chain or come all the way from the top of it.

Note: It is hard to bring change to more than just your team without the support of management. Even inside your team it is difficult if others are already set in their ways. For a new team in a new company a successful evangelist can often affect the forming policies even without the support of management simply by being a leader or creating the path of least resistance for others to follow. But in established company, see above.


Can "When" like keyword be used in scripted pipeline?

A Scripted Pipeline is literally a Groovy script, so you can just use if/else conditionals like you would in a normal Groovy script.

if (env.BRANCH_NAME == 'master') {
  // do steps here
} else {
  // do other steps here
}


NetworkPolicy: Why are pod selectors used over "service selectors"?

Pods is the deployment unit in Kubernetes and it is also the unit that gets a single unique IP address within the cluster.
You can access Pods without going via a Service. A Service is mostly there to get a "stable" address for an application, where the traffic is then load balanced to any Pod representing the Service. Both Pods and Services gets a DNS name in the cluster.
A Service has a selector to identify what Pods belong to the Service, but a Pod can receive traffic from multiple Services. In the same way a NetworkPolicy has a selector to select the Pods where it is applied.
Since a Pod is the smallest unit in Kubernetes, that gets its own unique network address it also makes sense to add the network security (e.g. Network Policies) at this level.


Bridged network adapter for Packer + VirtualBox?

With a bridged adapter, your VM would get its IP from the host machine's network DHCP server. This would allow it to be pinged from your internal network.

I'd also specify in your configuration that your cable is connected as so:

VBoxManage modifyvm "VM" --cableconnected1 on

Port forwarding in the VM's configuration would also work. It would allow access from the host machine's IP by just specifying the forwarded port.

To forward port 22 to 2222 you would do as so:

VBoxManage modifyvm "VM" --nic1 nat
VBoxManage modifyvm "VM" --natpf1 "guestssh,tcp,,2222,,22"
VBoxManage modifyvm "VM" --cableconnected1 on

If you want it to be accessible by the internet, you would need do port forwarding on your router to route access from your public ip:port to your internal ip:port.


How to update frontend servers behind a load balancer

What you have above seems like a reasonable update procedure, however you can also use persistence or a sticky session so that when a user goes to a web server, they will be routed to the same web server for the life of their browser session (or for the next hour, or whatever you decide to configure). This also allows for some A/B testing if you need to get user feedback on a new feature or just allow for the possibility of a bug (just pull server group B from the pool if an issue is discovered during A/B testing with that group).

There are some concerns you should be aware of with sticky sessions: Eg, source address persistence can be problematic if you have a thousand users behind a firewall - then you have a huge group that always goes to a single web node. Maybe you really don't want a cookie to do persistence, etc.

But, there really is no agreed "best practice" on this - the best practice is going to the the one that meets your business need in this case. You should just collect all of your options and choose the workflow that best meets your needs.


What are sources for configuration management best practices?

Well, that's pretty broad, mostly this can boils down to two options:

Policy based management
Promise Theory

In the first one you're describing the actions you want to be done to achieve the desired state, assuming previous state of the system or testing against it specifically, in the second you're describing what you expect as resulting state of the system and your tool should take care of doing what is necessary to achieve that.

A policy based management could be made of simple scripts, doing what you want, Promise Theory tools are inherently more complex (Ansible, Chef, Puppet, Salt) but they're mostly idempotent, this means they will only change something if the actual state doesn't match the desired state.

The only question which can be given a really definitive answer out of thin air without knowing the specifics of your application/service is to the second question:

Don't reinvent the wheel, there's serious softwares for configuration management nowadays that rewriting your own would be a road of pain for less functionalities than already existing in available softwares. There's of course a learning curve whatever path you choose, but the effort to learn a maintained piece is more worthy than the effort to write your own system and maintain it on top of the application configuration you'll have to manage with it.


ArgoCD Autosync based off of Docker Image `latest` Tag

This isn't really an ArgoCD thing. Keel is a good tool to help keep your images updated across the cluster and is super easy to implement. Basically, you'd just add some annotations to your manifest.
metadata:
  annotations:
    keel.sh/policy: force
    keel.sh/trigger: poll
    keel.sh/pollSchedule: "@every 10m"

It may be worth noting that if you are moving to run this in production, avoid the use of the latest tag. This is considered bad practice as it can burn you when breaking changes are pushed. Keel can handle that through an innate understanding of semver tags allowing you to update on minor version changes, but avoid major without testing, etc.